Data cleaning method, device, readable medium and electronic device
The data cleaning method identifies and cleans abnormal data points in ocean tidal power generation systems by calculating neighborhood ranges and outlier indices, enhancing data accuracy.
Patent Information
- Application Number
- CN202010934633.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-08
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2040-09-08
AI Technical Summary
Existing ocean tidal power generation systems face challenges with complex data sets containing inaccurate and abnormal data, which affect the accuracy of subsequent data analysis and processing results, lacking effective data cleaning technologies.
A data cleaning method that determines outlier data points by calculating neighborhood ranges, reachable densities, and outlier indices to identify and clean abnormal data.
Improves data accuracy by effectively screening and processing abnormal data points, providing a solid foundation for subsequent data analysis.
Smart Images
Figure CN114153825B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data analysis, and in particular to a data cleaning method, device, readable medium and electronic device. Background Art
[0002] Using ocean tidal energy to generate electricity is a renewable clean energy. Tidal ocean energy has many advantages such as never-exhaustion, high energy density, and strong predictability. Therefore, it has received extensive attention and development in recent years and is a hot spot in the field of new energy exploration.
[0003] In ocean tidal power generation projects, the Internet of Things monitoring system developed based on big data and artificial intelligence technology can be combined to better cope with the complex marine environment and comprehensively monitor the working status of the equipment.
[0004] It is understandable that the relevant data collected in the ocean tidal power generation platform is usually very complex and may be accompanied by a large amount of inaccurate and abnormal data, which will directly affect the accuracy of subsequent data analysis and processing results. The prior art lacks a targeted data cleaning technology to solve this technical problem. Summary of the invention
[0005] The present invention provides a data cleaning method, device, readable medium and electronic device, which realize the determination and cleaning of abnormal data by calculating and determining outlier data points, thereby improving the accuracy of data.
[0006] In a first aspect, the present invention provides a data cleaning method, comprising:
[0007] Determine the neighborhood range of the target data point according to a preset range parameter; and determine a neighborhood data set within the neighborhood range; the neighborhood data set includes at least one neighborhood data point;
[0008] Determining a first reachable density of the target data point according to the target data point and each of the neighborhood data points;
[0009] Determining an outlier index of the target data point according to the first reachable density;
[0010] When the outlier index meets a preset condition, the target data point is determined as an outlier data point, and the outlier data point is cleaned.
[0011] Preferably, the target data point is P, and the range parameter is k, then determining the neighborhood range of the target data point according to the preset range parameter, and determining the neighborhood data set within the neighborhood range includes:
[0012] Determine a first data point O0, and use the distance from the target data point P to the first data point O0 as a first neighborhood distance;
[0013] Determine a circular range with the target data point P as the center and the first neighborhood distance as the radius;
[0014] A circular range is used as the neighborhood range, and the number of second data points in the neighborhood range is at most k-1;
[0015] Determine the first data point and the second data point as neighborhood data points; and establish a neighborhood data set based on the neighborhood data points;
[0016] Wherein, k is a positive integer.
[0017] Preferably, determining the first reachable density of the target data point according to the target data point and each of the neighborhood data points comprises:
[0018] Determine the reachable distance between each of the neighborhood data points and the target data point;
[0019] A first reachable density of the target data point is determined according to each of the reachable distances.
[0020] Preferably, the neighborhood data point is 0, and the determining of the reachable distance between the neighborhood data point and the target data point includes:
[0021] Determine the distance d(P, O) from the neighborhood data point O to the target data point P;
[0022] Determine the second neighborhood distance d corresponding to the neighborhood data point O k (O);
[0023] Then the reachable distance between the neighborhood data point and the target data point, reach_dist(P, O), is:
[0024] reach_dist(P,O)=max{d(P,O),d k (O)}.
[0025] Preferably, determining the first reachable density of the target data point according to each reachable distance comprises:
[0026]
[0027] Wherein, lrd(P) represents the first reachable density of the target data point; N k (P) represents the neighborhood data set, |N k (P)| represents the number of data points in the neighborhood data set; O represents any neighborhood data point; Represents the sum of the reachable distances between each of the neighborhood data points and the target data point.
[0028] Preferably, determining the outlier index of the target data point according to the first reachable density comprises:
[0029] Determine a second reachable density corresponding to each of the neighborhood data points;
[0030] An outlier index of the target data point is determined according to the first reachable density and each of the second reachable densities.
[0031] Preferably, the outlier data points are cleaned up including:
[0032] The outlier data points are deleted, regarded as null values, corrected or ignored.
[0033] In a second aspect, the present invention provides a data cleaning device, comprising:
[0034] A neighborhood determination module is used to determine the neighborhood range of the target data point according to a preset range parameter; and determine a neighborhood data set within the neighborhood range; the neighborhood data set includes at least one neighborhood data point;
[0035] A reachable density calculation module, used to determine a first reachable density of the target data point according to the target data point and each of the neighborhood data points;
[0036] an outlier index calculation module, configured to determine an outlier index of the target data point according to the first reachable density;
[0037] A cleaning module is used to determine the target data point as an outlier data point and perform cleaning processing on the outlier data point when the outlier index meets a preset condition.
[0038] In a third aspect, the present invention provides a readable medium, comprising execution instructions. When a processor of an electronic device executes the execution instructions, the electronic device executes the data cleaning method as described in any one of the first aspects.
[0039] In a fourth aspect, the present invention provides an electronic device, comprising a processor and a memory storing execution instructions, wherein when the processor executes the execution instructions stored in the memory, the processor executes the data cleaning method as described in any one of the first aspects.
[0040] The present invention provides a data cleaning method, device, readable medium and electronic device, which determine the neighborhood range of a target data point and the neighborhood data points, and then calculate the outlier index of the target data point; judge whether the target data point is abnormal according to the outlier index; when the outlier index meets the preset conditions, the target data point is determined as an outlier data point, and the outlier data point is cleaned; thereby, the screening and processing of abnormal data points are realized, the accuracy of the data is improved, and a good foundation is provided for subsequent data analysis.
[0041] The further effects of the above-mentioned non-conventional preferred manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the embodiments of the present invention or the existing technical solutions, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0043] Figure 1 A flowchart of a data cleaning method provided by an embodiment of the present invention;
[0044] Figure 2 A schematic diagram of the domain scope in a data cleaning method provided by an embodiment of the present invention;
[0045] Figure 3 A schematic diagram of the structure of a data cleaning device provided by an embodiment of the present invention;
[0046] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0047] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0048] The relevant data collected in the ocean tidal power generation project is usually very complex and may be accompanied by a large amount of inaccurate and abnormal data, which will directly affect the accuracy of the subsequent data analysis and processing results. The prior art lacks a targeted data cleaning technology to solve this technical problem. The present invention provides a data cleaning method, device, readable medium and electronic device, which can determine and clean abnormal data by calculating and determining outlier data points, thereby improving the accuracy of the data.
[0049] See also Figure 1 FIG. 1 is a specific embodiment of the data cleaning method provided by the present invention. In this embodiment, the method specifically includes the following steps:
[0050] Step 101: determine a neighborhood range of a target data point according to a preset range parameter; and determine a neighborhood data set within the neighborhood range; the neighborhood data set includes at least one neighborhood data point.
[0051] The target data point is the data point targeted in this embodiment. The method in this embodiment is to determine whether the target data point is abnormal, and to perform further cleaning processing if it is abnormal. In this embodiment, it is believed that the distribution of data points usually has a more obvious regularity, and normal data points are often densely distributed within a certain range. On the contrary, data points with "outlier" characteristics are most likely to be abnormal. In this embodiment, the target data point is analyzed based on this principle. In this embodiment, the target data point can be represented by P.
[0052] In this embodiment, a range parameter can be determined, and the range parameter can be represented by a positive integer k. The specific value of the range parameter can be set according to needs and experience, and will not be repeated here. The neighborhood range of the target data point can be determined by using the range parameter.
[0053] Specifically, the kth data point in order of distance from the target data point P from nearest to farthest can be determined as the first data point O0. And the distance d(P, O0) from the target data point P to the first data point O0 is taken as the first neighborhood distance. Of course, if there may be other data points whose distance from the target data point P is equal to the first neighborhood distance, then the number of first data points can be multiple. The first neighborhood distance can also be the k distance of the target data point P, and the k distance can be expressed as d k (P) is used to indicate this.
[0054] Then, a circular range is determined with the target data point P as the center and the first neighborhood distance d(P, O0) as the radius. This circular range is the neighborhood range corresponding to the target data point P, such as Figure 2As shown. It can be understood that the neighborhood range will include k-1 second data points that are closer to the target data point (relative to the first data point). The first data point and the second data point are determined as neighborhood data points; and a neighborhood data set is established based on the neighborhood data points. In this embodiment, the neighborhood data set is N k (P) indicates. |N k (P)| represents the number of data points in the neighborhood data set. When the number of the first data point is 1, |N k (P)|=k; otherwise, when the number of the first data point is greater than 1, |N k (P)|>k.
[0055] In this embodiment, it is assumed that k = 5. Figure 2 It can be seen that the neighborhood range includes 4 (k-1) second data points O1-O4, and the edge of the neighborhood range has the first data point O0. At this time, the neighborhood data set includes 5 neighborhood data points O0-O4.
[0056] Step 102: Determine a first reachable density of the target data point based on the target data point and each of the neighborhood data points.
[0057] In this embodiment, the reachable distance between each of the neighborhood data points and the target data point is determined; and the first reachable density of the target data point is determined according to each of the reachable distances.
[0058] Assume that O represents Figure 2 Any one of the five neighborhood data points O0-O4 in the figure. Then the distance d(P, O) from the neighborhood data point O to the target data point P can be determined, and the second neighborhood distance d corresponding to the neighborhood data point O can be determined. k (O).
[0059] It should be noted that the calculation method of the second neighborhood distance corresponding to the neighborhood data point O is the same as that of the first neighborhood distance. That is, when the value of the range parameter k is set unchanged, the neighborhood range corresponding to the neighborhood data point O is determined in the same way; and the second neighborhood distance d corresponding to the neighborhood data point O is determined at the same time. k (O), which is the k distance corresponding to the neighborhood data point O.
[0060] In fact, the distribution density of data points near the target data point P and the neighborhood data point O may be different. Therefore, when the density of data points near the neighborhood data point O is close to or less than the density of data points near the target data point P, the target data point P may be within the neighborhood range of the neighborhood data point O. If the density of data points near the neighborhood data point O is greater than the density of data points near the target data point P, the target data point P may not be within the neighborhood range of the neighborhood data point O. When the target data point P is within the neighborhood range of the neighborhood data point O, the distance d(P, O) between the two will not be greater than the second neighborhood distance d corresponding to the neighborhood data point O. k (O). On the contrary, when the target data point P may not be within the neighborhood range of the neighborhood data point O, the distance d(P, O) between the two will be greater than the second neighborhood distance d corresponding to the neighborhood data point O. k (O).
[0061] In this embodiment, the calculation formula of the reachable distance reach_dist(P, O) between the neighborhood data point O and the target data point P is:
[0062] reach_dist(P,O)=max{d(P,O),d k (O)}.
[0063] The first reachable density of the target data point is determined according to the reachable distance of each neighborhood data point, that is, the inverse of the average of each reachable distance is calculated. The calculation formula is as follows:
[0064]
[0065] Wherein, lrd(P) represents the first reachable density of the target data point; N k (P) represents the neighborhood data set, |N k (P)| represents the number of data points in the neighborhood data set; O represents any neighborhood data point; Represents the sum of the reachable distances between each of the neighborhood data points and the target data point.
[0066] According to the calculation principle of reachable distance, it can be understood that when the density of data points near the neighborhood data point O is close to or less than the density of data points near the target data point P, the target data point P is within the neighborhood range of the neighborhood data point O, that is, d(P, O)≤d k (O), then reach_dist(P, O) = d k (O). On the contrary, when the density of data points near the neighborhood data point O is greater than the density of data points near the target data point P, the target data point P is not within the neighborhood range of the neighborhood data point O, that is, d(P, O)>d k (O), reach_dist(P, O)=d(P, O). ,
[0067] It can be inferred that when the density of data points near the target data point P is large, the value of reach_dist(P, O) is small. The value is also small, so the value of lrd(P) is large. On the contrary, if the density of data points near the target data point P is small, the value of reach_dist(P, O) is large. The value is also large, so the value of lrd(P) is small. When the value of lrd(P) is less than a certain level, it means that the density of data points near the target data point P is too low, indicating that the target data point P may have the characteristics of "outliers".
[0068] Step 103: Determine an outlier index of the target data point according to the first reachable density.
[0069] In this step, the outlier index of the target data point will be determined based on the first reachable density, so as to specifically measure whether the density of data points near the target data point P is too low, so that the target data point P has the characteristic of "outlier". Specifically, in this embodiment, the second reachable density corresponding to each of the neighborhood data points can be determined; based on the first reachable density and each of the second reachable densities, the outlier index of the target data point is determined.
[0070] The calculation method of the second reachable density is similar to that of the first reachable density, and will not be repeated here. In this embodiment, the second reachable density of the neighborhood data point O is represented as lrd(O). The sum of the second reachable densities corresponding to the neighborhood data points can be represented as In this embodiment, the outlier index of the target data point P is expressed as LOF(P), which is the ratio of the average value of the second reachable density to the first reachable density. The calculation formula is as follows:
[0071]
[0072] If the outlier index is less than or close to 1, it means that the density of data points near the target data point P is normal. If the outlier index is significantly greater than 1, it means that the density of data points near the target data point P is low, and the target data point P has the characteristics of "outliers".
[0073] Step 104: When the outlier index meets a preset condition, the target data point is determined as an outlier data point, and the outlier data point is cleaned.
[0074] In this embodiment, a corresponding index threshold can be set for the outlier index, for example, set to 1.3. When the outlier index is greater than the index threshold, it means that the target data point P has a very obvious "outlier" characteristic and should be determined as an outlier data point, that is, an abnormal data point.
[0075] In view of this situation, in this embodiment, the outlier data points can be cleaned. Specifically, they can be deleted, regarded as null values, corrected or ignored. This is not limited in this embodiment, and in actual situations, an appropriate data cleaning method can be selected according to needs.
[0076] It can be seen from the above technical scheme that the beneficial effects of this embodiment are: by determining the neighborhood range of the target data point and determining the neighborhood data points, the outlier index of the target data point is calculated; whether the target data point is abnormal is judged according to the outlier index; when the outlier index meets the preset conditions, the target data point is determined as an outlier data point, and the outlier data point is cleaned; thereby, the screening and processing of abnormal data points are realized, the accuracy of the data is improved, and a good foundation is provided for subsequent data analysis.
[0077] like Figure 3 The figure shows a specific embodiment of the data cleaning device of the present invention. The device described in this embodiment is used to perform Figure 1 The physical device of the method. Its technical solution is essentially consistent with the above embodiment, and the corresponding description in the above embodiment is also applicable to this embodiment. The device in this embodiment includes:
[0078] The neighborhood determination module 301 is used to determine the neighborhood range of the target data point according to a preset range parameter; and determine a neighborhood data set within the neighborhood range; the neighborhood data set includes at least one neighborhood data point.
[0079] The reachable density calculation module 302 is used to determine a first reachable density of the target data point according to the target data point and each of the neighborhood data points.
[0080] The outlier index calculation module 303 is used to determine the outlier index of the target data point according to the first reachable density.
[0081] The cleaning module 304 is used to determine the target data point as an outlier data point and perform cleaning processing on the outlier data point when the outlier index meets a preset condition.
[0082] Figure 4 : is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage, etc. Of course, the electronic device may also include hardware required for other services.
[0083] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0084] The memory is used to store execution instructions. Specifically, the execution instructions are computer programs that can be executed. The memory may include internal memory and non-volatile memory, and provides execution instructions and data to the processor.
[0085] In one possible implementation, the processor reads the corresponding execution instruction from the non-volatile memory into the memory and then runs it, and can also obtain the corresponding execution instruction from other devices to form a data cleaning device at the logical level. The processor executes the execution instruction stored in the memory to implement the data cleaning method provided in any embodiment of the present invention by executing the execution instruction.
[0086] As described above, the present invention Figure 3 The method performed by the data cleaning device provided in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or instructions in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present invention can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0087] The steps of the method disclosed in the embodiment of the present invention can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0088] The embodiment of the present invention further provides a readable medium, which stores an execution instruction. When the stored execution instruction is executed by a processor of an electronic device, the electronic device can execute the data cleaning method provided in any embodiment of the present invention, and is specifically used to execute the following Figure 1 or Figure 2 The method shown.
[0089] The electronic device described in the above embodiments may be a computer.
[0090] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may be implemented in the form of a complete hardware embodiment, a complete software embodiment, or a combination of software and hardware.
[0091] The various embodiments of the present invention are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0092] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0093] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.
Claims
1. A data cleaning method, It is characterized in that include: Determine the neighborhood range of the target data point according to the preset range parameters; and determining a neighborhood data set within the neighborhood range; The neighborhood data set includes at least one neighborhood data point; the target data point is relevant data collected in the ocean tidal energy power generation project, and the value of the preset range parameter is set according to the project requirements and experience of the ocean tidal energy power generation; Determining a first reachable density of the target data point according to the target data point and each of the neighborhood data points; Determining an outlier index of the target data point according to the first reachable density; When the outlier index meets the preset condition, the target data point is determined as an outlier data point, and the outlier data point is cleaned; Wherein, the target data point is P, and the range parameter is k, then determining the neighborhood range of the target data point according to the preset range parameter, and determining the neighborhood data set within the neighborhood range includes: Determine the first data point O 0 , and move the target data point P to the first data point O 0 The distance is taken as the first neighbor distance; Determine a circular range with the target data point P as the center and the first neighborhood distance as the radius; A circular range is used as the neighborhood range, and the number of second data points in the neighborhood range does not exceed k-1; Determine the first data point and the second data point as neighborhood data points; and establish a neighborhood data set based on the neighborhood data points; Wherein, k is a positive integer.
2. The method according to claim 1, It is characterized in that Determining a first reachable density of the target data point according to the target data point and each of the neighborhood data points includes: Determine the reachable distance between each of the neighborhood data points and the target data point; A first reachable density of the target data point is determined according to each of the reachable distances.
3. The method according to claim 2, It is characterized in that The neighborhood data point is 0, then determining the reachable distance between the neighborhood data point and the target data point includes: Determine the distance d(P, O) from the neighborhood data point O to the target data point P; Determine the second neighborhood distance d corresponding to the neighborhood data point O k (O); Then the reachable distance reach_dist(P, O) between the neighborhood data point and the target data point is, reach_dist(P,O)=max{ d(P,O),d k (O)}。 4. The method according to claim 3, It is characterized in that Determining the first reachable density of the target data point according to each reachable distance comprises: in, A first reachable density representing the target data point; represents the neighborhood data set, represents the number of data points in the neighborhood data set; O represents any neighborhood data point; Represents the sum of the reachable distances between each of the neighborhood data points and the target data point.
5. The method according to claim 4, It is characterized in that Determining the outlier index of the target data point according to the first reachable density includes: Determine a second reachable density corresponding to each of the neighborhood data points; An outlier index of the target data point is determined according to the first reachable density and each of the second reachable densities.
6. The method according to any one of claims 1 to 5, It is characterized in that The outlier data point cleaning process includes: The outlier data points are deleted, regarded as null values, corrected or ignored.
7. A data cleaning device, It is characterized in that include: A neighborhood determination module, used to determine the neighborhood range of the target data point according to preset range parameters; and determining a neighborhood data set within the neighborhood range; The neighborhood data set includes at least one neighborhood data point; the target data point is relevant data collected in the ocean tidal energy power generation project, and the value of the preset range parameter is set according to the project requirements and experience of the ocean tidal energy power generation; A reachable density calculation module, used to determine a first reachable density of the target data point according to the target data point and each of the neighborhood data points; an outlier index calculation module, configured to determine an outlier index of the target data point according to the first reachable density; A cleaning module, used for determining the target data point as an outlier data point and performing cleaning processing on the outlier data point when the outlier index meets a preset condition; Wherein, the target data point is P, the range parameter is k, and the neighborhood determination module is specifically used to: determine the first data point O 0 , and move the target data point P to the first data point O 0 as the first neighborhood distance; determine a circular range with the target data point P as the center and the first neighborhood distance as the radius; take the circular range as the neighborhood range, and satisfy that the number of second data points in the neighborhood range does not exceed k-1; determine the first data point and the second data point as neighborhood data points; and establish a neighborhood data set based on the neighborhood data points; wherein k is a positive integer.
8. A readable medium comprising an execution instruction. When a processor of an electronic device executes the execution instruction, the electronic device executes the data cleaning method as described in any one of claims 1 to 6.
9. An electronic device, comprising a processor and a memory storing execution instructions, wherein when the processor executes the execution instructions stored in the memory, the processor executes the data cleaning method as described in any one of claims 1 to 6.