Data quality improvement method for data processing

Abstract

The invention relates to the technical field of data quality improvement, and particularly discloses a data quality improvement method for data processing, which comprises the following steps: S1, acquiring all historical data from a database, classifying the historical data, and selecting a plurality of cleaning methods; for each cleaning method, calculating an efficiency value and a quality value of each cleaning method on each type of historical data; s2, calculating a matching degree between each cleaning method and each type of historical data based on the efficiency value and the quality value; selecting the cleaning method with the highest matching degree as the optimal cleaning method of the historical data; s3, when the user stores the current data to the database, firstly determining the type of the current data; cleaning the current data by using the determined optimal cleaning method; by analyzing and comparing the efficiency and quality of different cleaning methods, the most suitable cleaning strategy is selected for different types of historical data, so that the quality of the whole data is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data quality improvement, and in particular to a data quality improvement method for data processing. Background Art

[0002] In the real world, there are many types of data, including single structured data (such as tabular data in a database), semi-structured data (such as data in JSON, XML and other formats) and unstructured data (such as text, images, videos, etc.); and there are also a variety of data cleaning methods, including binning, regression and clustering.

[0003] In the prior art, when a database cleans the stored data, it often only uses one method to clean the data; however, since there are many types of data and the cleaning effects and efficiencies of cleaning methods for different types of data vary significantly, the cleaning quality is not high; and although some human-computer interaction methods can improve the cleaning quality, their efficiency is relatively low. Summary of the invention

[0004] The purpose of the present invention is to provide a method for improving data quality for data processing to solve the above technical problems.

[0005] The purpose of the present invention can be achieved through the following technical solutions: A method for improving data quality for data processing comprises the following steps: Step S1: Acquire all historical data stored in the database, divide the historical data into several types; Select several data cleaning methods, and for any one of the cleaning methods, use the cleaning method to clean various types of historical data to obtain the efficiency value of each type of historical data. And the quality value M=m / num, where n is the total number of historical data of the same type, t i represents the time consumed by the cleaning method to clean the i-th historical data in the same type of historical data, i∈[1,n] and i is a positive integer, m is the number of abnormal data cleaned by the cleaning method in the same type of historical data, and num is the total number of abnormal data in the same type of historical data; Step S2: According to the efficiency value and quality value of each type of historical data when the cleaning method is used to clean, the matching degree between the cleaning method and the i-th type of historical data is obtained. , where f i is the efficiency value of the i-th type of historical data, M i is the quality value of the i-th category of historical data, f ave It represents the average efficiency value of each type of historical data under the same cleaning method, Mave It indicates the average quality value of each type of historical data under the same cleaning method; Then, a matching degree set J of the cleaning method is obtained, a maximum value Max(J) in the set J is selected, the historical data type corresponding to the maximum value Max(J) is obtained, and the cleaning method is recorded as the optimal cleaning method for the historical data type; Step S3: When the user stores the current data in the database, the type of the current data is determined and recorded as the current data type; the best cleaning method for the current data type is obtained, and the current data is cleaned using the best cleaning method.

[0006] As a further solution of the present invention: the process of dividing the historical data includes: The historical data is converted into data points to obtain n data points, and the n data points are placed in a spatial coordinate system to select a number of spherical areas, and k clusters are selected according to the data point density of each spherical area, that is, k types are obtained.

[0007] As a further solution of the present invention: the process of obtaining the data point density includes: An initial region radius R is set, an interval threshold r is set, and the initial region radius R is increased and decreased according to the interval threshold to obtain a plurality of region radius gradients; a region radius set is obtained from all region radius gradients and the initial region radius; A center point is randomly selected in the spatial coordinate system, and the elements in the region radius set are used as the radius to obtain a number of spherical regions, and the data point density in the spherical region is obtained. , where P e represents the e-th data point in the spherical region, P x Indicates that the spherical region is P e The xth data point outside, |P e -P x | indicates P e With P x The distance between them, N represents the total number of data points in the spherical area, e∈[1,N] and e is a positive integer, x∈[1,N-1] and x is a positive integer.

[0008] As a further solution of the present invention: the process of dividing the historical data further includes: According to the data point density of each spherical area, a data point density set is obtained. In the data point density set, elements whose data point density is greater than a preset threshold are selected and recorded as candidate elements; and the element with the largest corresponding area radius is selected from all the candidate elements, and the spherical area corresponding to the element is recorded as a cluster; All data points in the cluster are excluded, new center points are continuously selected from the remaining data points in the spatial coordinate system, and new clusters are selected according to the above steps, and so on, to obtain k clusters.

[0009] As a further solution of the present invention: the abnormal data includes missing values, duplicate values ​​and abnormal values; the abnormal data in the historical data is divided in advance based on manual or automated cleaning tools.

[0010] As a further solution of the present invention: if m>num occurs, the quality value at this time is directly recorded as 1.

[0011] As a further solution of the present invention: when the maximum values ​​in the set J are several equal values, the mass values ​​corresponding to the maximum values ​​are obtained, and the maximum value with the largest mass value is selected and recorded as Max(J).

[0012] As a further solution of the present invention: the process of determining the type of the current data includes: In the spatial coordinate system, the coordinates of the center point in each cluster are obtained, recorded as the center coordinates; the current data is converted into a data point in the spatial coordinate system, and its coordinates are obtained, recorded as the current coordinates; among all the center coordinates, the center coordinates closest to the current coordinates are selected, recorded as similar center coordinates; and the cluster corresponding to the similar center coordinates is obtained, recorded as the type of the current data.

[0013] Beneficial effects of the present invention: The present invention can effectively identify and remove abnormal data through a variety of data cleaning methods, thereby improving the accuracy and reliability of data sets; select the most appropriate cleaning method according to different types of historical data to ensure that each type of data can be processed most effectively; by calculating the efficiency value of each cleaning method, the method that performs best on a specific type of data can be selected to reduce unnecessary computing overhead; when the user stores the current data, the data type is automatically determined and the best cleaning method is applied, reducing the need for manual intervention and improving the processing speed; by comprehensively considering the efficiency value and the quality value to calculate the matching degree, it is ensured that the selected cleaning method is not only fast but also efficient; dynamically adjust the best cleaning method according to the actual cleaning effect to adapt to the changes in data characteristics; the core of the present invention is to find the cleaning method that is most suitable for a specific type of historical data through experiments and evaluation of different data cleaning technologies. By comparing the performance of different methods in terms of efficiency and quality, the best cleaning strategy can be selected to improve the quality of the overall data; this data quality improvement method greatly improves the efficiency and quality of data processing through intelligent matching and automated processing procedures. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present invention will be further described below in conjunction with the accompanying drawings.

[0015] Figure 1 It is a flow chart of a method for improving data quality for data processing according to the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] See also Figure 1 As shown, the present invention is a method for improving data quality for data processing, comprising the following steps: Step S1: Acquire all historical data stored in the database, divide the historical data into several types; Select several data cleaning methods, and for any one of the cleaning methods, use the cleaning method to clean various types of historical data to obtain the efficiency value of each type of historical data. And the quality value M=m / num, where n is the total number of historical data of the same type, t i represents the time consumed by the cleaning method to clean the i-th historical data in the same type of historical data, i∈[1,n] and i is a positive integer, m is the number of abnormal data cleaned by the cleaning method in the same type of historical data, and num is the total number of abnormal data in the same type of historical data; It can be understood that the cleaning methods include binning, regression and clustering; the efficiency value is calculated by taking the average cleaning time of the same type of historical data, and the quality value is evaluated by calculating the proportion of abnormal data after cleaning; It is worth noting that the time consumed for cleaning a single data is often very short and difficult to measure accurately. To facilitate calculation, several data can be selected as a group, and the time consumed for cleaning each group of data can be obtained. Then, the time average is divided by the number of data in a group to obtain the time required for cleaning a data. As a preferred embodiment of the present invention, the process of dividing the historical data includes: The historical data is converted into data points to obtain n data points, the n data points are placed in a spatial coordinate system to select a number of spherical regions, and k clusters are selected according to the density of data points in each spherical region, that is, k types are obtained; The process of obtaining the data point density includes: An initial region radius R is set, an interval threshold r is set, and the initial region radius R is increased and decreased according to the interval threshold to obtain a plurality of region radius gradients; a region radius set is obtained from all region radius gradients and the initial region radius; A center point is randomly selected in the spatial coordinate system, and the elements in the region radius set are used as the radius to obtain a number of spherical regions, and the data point density in the spherical region is obtained. , where P e represents the e-th data point in the spherical region, P x Indicates that the spherical region is P e The xth data point outside, |P e -P x | indicates P e With P x The distance between them, N represents the total number of data points in the spherical region, e∈[1,N] and e is a positive integer, x∈[1,N-1] and x is a positive integer; The process of dividing the historical data further includes: According to the data point density of each spherical area, a data point density set is obtained. In the data point density set, elements whose data point density is greater than a preset threshold are selected and recorded as candidate elements; and the element with the largest corresponding area radius is selected from all the candidate elements, and the spherical area corresponding to the element is recorded as a cluster; Exclude all data points in the cluster, continue to select new center points from the remaining data points in the spatial coordinate system, and select new clusters according to the above steps, and so on, to obtain k clusters; It is worth noting that in the process of converting the historical data into data points, different types of data have different conversion methods, such as the following: For numerical data, each feature or variable can be directly used as a dimension on the coordinate axis. For example, if the data set contains three features (age, income, and expenditure), each data point can be represented as a point (x, y, z) in three-dimensional space, where x represents age, y represents income, and z represents expenditure. For categorical data, it is necessary to convert it into numerical data through encoding. Common methods include One-Hot Encoding and Label Encoding. For example, color features can be represented by One-Hot Encoding, where red, blue, and green are encoded as (1,0,0), (0,1,0), and (0,0,1) respectively. Text data usually needs to be converted into numerical features through natural language processing technology; image data can be directly represented by pixel values, and the color value of each pixel (usually RGB) can be used as the value of a coordinate axis; in addition, deep learning models such as convolutional neural networks (CNNs) can be used to extract higher-level features, such as edges, textures, and shapes, which will not be elaborated here; It is worth noting that when selecting the center point, data points located at the edge or discrete area of ​​the spatial coordinate system are generally not included; this is because the points located at the edge may not represent the typical characteristics of the entire data set, resulting in the clustering results being biased towards a specific part of the data set rather than the global distribution; in addition, the edge points may not be related to the main trend in the data set; the points in the discrete area are usually outliers or noise points in the data, which may distort the shape and size of the cluster and lead to incorrect clustering decisions; selecting the center point that can represent most of the points in the data set can reduce the deviation of the clustering and improve the accuracy of the clustering; It should be noted that the process of selecting the center point includes: Mark each data point with a black dot in the spatial coordinate system, and divide the spatial coordinate system into a number of cube grids to obtain a number of cubes; obtain a three-dimensional screenshot of each cube, wherein the three-dimensional screenshot is three adjacent faces on the cube compared to the same vertex; and obtain a mark ratio Pro=Ma / Ma´ in the three-dimensional screenshot, wherein Ma is the area occupied by black in the three-dimensional screenshot, and Ma´ is the total area of ​​the three-dimensional screenshot; obtain the mark ratio of each cube, and sort them from large to small, and according to the sorted cubes, randomly select a data point in the cube as the center point for analysis and obtain a cluster; It should be noted that the process of obtaining the data point density is to obtain any data point in the spherical region, and obtain all data points in the spherical region except the data point, which are recorded as the remaining data points; obtain the sum of the distances between the data point and the remaining data points, which is recorded as the cumulative distance, that is, ; and obtain the average value of the distance accumulation of each data point in the spherical area, which is the data point density; the preset threshold is recorded as r t, the candidate element R s ={ r s | r s > r t}, where r s Represents the data point density of the sth element, and selects the element with the largest area radius from the candidate elements Rmax=Max(R s ); As a preferred embodiment of the present invention, the abnormal data includes missing values, duplicate values ​​and abnormal values; the abnormal data in the historical data is divided in advance based on manual or automated cleaning tools; It is understood that the missing values ​​are data points that lack the required information or values ​​in some records in the data set, the duplicate values ​​are data points that have exactly the same records or values ​​in the data set, and the outliers are data points that do not conform to the expected range or pattern, which may be caused by errors, noise or other factors; As a preferred embodiment of the present invention, if m>num occurs, the quality value at this time is directly recorded as 1; It is worth noting that when using the cleaning method to clean the data, normal data is often mistakenly judged as abnormal data. In this case, m>num is likely to occur. In this case, the quality value is directly recorded as 1; Step S2: According to the efficiency value and quality value of each type of historical data when the cleaning method is used to clean, the matching degree between the cleaning method and the i-th type of historical data is obtained. , where f i is the efficiency value of the i-th type of historical data, M i is the quality value of the i-th category of historical data, f ave It represents the average efficiency value of each type of historical data under the same cleaning method, M ave It indicates the average quality value of each type of historical data under the same cleaning method; Then, a matching degree set J of the cleaning method is obtained, a maximum value Max(J) in the set J is selected, the historical data type corresponding to the maximum value Max(J) is obtained, and the cleaning method is recorded as the optimal cleaning method for the historical data type; It should be noted that, assuming there are n types of historical data, the efficiency value and quality value of each type are {(f1, M1), (f2, M2), ..., (f n , M n )}, the average efficiency value and the average quality value are respectively and ; As a preferred embodiment of the present invention, when the maximum value in the set J is a plurality of equal values, the quality values ​​corresponding to the maximum values ​​are obtained, and the maximum value with the largest quality value is selected and recorded as Max(J); It is worth noting that the quality value is more important than the efficiency value. When there are several equal values ​​of the maximum value in the set J, it is necessary to further compare the quality values ​​corresponding to these maximum values; Step S3: When the user stores the current data in the database, the type of the current data is determined and recorded as the current data type; the best cleaning method for the current data type is obtained, and the current data is cleaned using the best cleaning method; As a preferred embodiment of the present invention, the process of determining the type of the current data includes: In the spatial coordinate system, the coordinates of the center points in each cluster are obtained, recorded as the center coordinates; the current data is converted into data points in the spatial coordinate system, and its coordinates are obtained, recorded as the current coordinates; among all the center coordinates, the center coordinates closest to the current coordinates are selected, recorded as similar center coordinates; and the cluster corresponding to the similar center coordinates is obtained, recorded as the type of the current data; It can be understood that in the spatial coordinate system, first obtain the coordinates of the center point in each cluster, recorded as the center coordinates, which represent the typical positions of each cluster; convert the current data into a point in the spatial coordinate system, and obtain the coordinates of this point, recorded as the current coordinates; this step is to represent the current data as a specific position in space; among all the center coordinates, find the center coordinate closest to the current coordinate, recorded as the similar center coordinate; this step is to determine which cluster the current data is closest to.

[0018] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A method for improving data quality for data processing, characterized in that: The following steps are involved: Step S1: Acquire all historical data stored in the database, divide the historical data into several types; Select several data cleaning methods, and for any one of the cleaning methods, use the cleaning method to clean various types of historical data to obtain the efficiency value of each type of historical data. And the quality value M=m / num, where n is the total number of historical data of the same type, t i represents the time consumed by the cleaning method to clean the i-th historical data in the same type of historical data, i∈[1,n] and i is a positive integer, m is the number of abnormal data cleaned by the cleaning method in the same type of historical data, and num is the total number of abnormal data in the same type of historical data; Step S2: According to the efficiency value and quality value of each type of historical data when the cleaning method is used to clean, the matching degree between the cleaning method and the i-th type of historical data is obtained. , where f i is the efficiency value of the i-th type of historical data, M i is the quality value of the i-th category of historical data, f ave It represents the average efficiency value of each type of historical data under the same cleaning method, M ave It indicates the average quality value of each type of historical data under the same cleaning method; Then, a matching degree set J of the cleaning method is obtained, a maximum value Max(J) in the set J is selected, the historical data type corresponding to the maximum value Max(J) is obtained, and the cleaning method is recorded as the optimal cleaning method for the historical data type; Step S3: When the user stores the current data in the database, the type of the current data is determined and recorded as the current data type; the best cleaning method for the current data type is obtained, and the current data is cleaned using the best cleaning method.

2. A method for improving data quality for data processing according to claim 1, characterized in that: In step S1, the process of dividing the historical data includes: The historical data is converted into data points to obtain n data points, and the n data points are placed in a spatial coordinate system to select a number of spherical areas, and k clusters are selected according to the data point density of each spherical area, that is, k types are obtained.

3. A method for improving data quality for data processing according to claim 2, characterized in that: In step S1, the process of obtaining the data point density includes: An initial region radius R is set, an interval threshold r is set, and the initial region radius R is increased and decreased according to the interval threshold to obtain a plurality of region radius gradients; a region radius set is obtained from all region radius gradients and the initial region radius; A center point is randomly selected in the spatial coordinate system, and the elements in the region radius set are used as the radius to obtain a number of spherical regions, and the data point density in the spherical region is obtained. , where P e represents the e-th data point in the spherical region, P x Indicates that the spherical region is divided by P e The xth data point outside, |P e -P x | indicates P e With P x The distance between them, N represents the total number of data points in the spherical area, e∈[1,N] and e is a positive integer, x∈[1,N-1] and x is a positive integer.

4. The method for improving data quality for data processing according to claim 3, characterized in that: In step S1, the process of dividing the historical data further includes: According to the data point density of each spherical area, a data point density set is obtained. In the data point density set, elements whose data point density is greater than a preset threshold are selected and recorded as candidate elements; and the element with the largest corresponding area radius is selected from all the candidate elements, and the spherical area corresponding to the element is recorded as a cluster; All data points in the cluster are excluded, new center points are continuously selected from the remaining data points in the spatial coordinate system, and new clusters are selected according to the above steps, and so on, to obtain k clusters.

5. The method for improving data quality for data processing according to claim 1, characterized in that: In step S1, the abnormal data includes missing values, duplicate values ​​and abnormal values; the abnormal data in the historical data is divided in advance based on manual or automated cleaning tools.

6. The method for improving data quality for data processing according to claim 1, characterized in that: In step S1, if m>num occurs, the quality value at this time is directly recorded as 1.

7. The method for improving data quality for data processing according to claim 1, characterized in that: In step S2, when the maximum values ​​in the set J are several equal values, the quality values ​​corresponding to the maximum values ​​are obtained, and the maximum value with the largest quality value is selected and recorded as Max(J).

8. The method for improving data quality for data processing according to claim 3, characterized in that: In step S3, the process of determining the type of the current data includes: In the spatial coordinate system, the coordinates of the center point in each cluster are obtained, recorded as the center coordinates; the current data is converted into a data point in the spatial coordinate system, and its coordinates are obtained, recorded as the current coordinates; among all the center coordinates, the center coordinates closest to the current coordinates are selected, recorded as similar center coordinates; and the cluster corresponding to the similar center coordinates is obtained, recorded as the type of the current data.

Images