Data recovery method and device
By standardizing the transformation and density clustering of business data, identifying data clusters and performing attribute correlation analysis, and generating attribute correlation diagrams and clusters, the problem of inaccurate data repair in the existing technology is solved, and efficient and accurate data repair is achieved .
Patent Information
- Application Number
- CN202310597632.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-25
- Publication Date
- 2024-11-26
AI Technical Summary
In the prior art, when the time sequence relationship of business data is not obvious or the attribute constraint relationship is not obvious, the data repair results are inaccurate and the repair efficiency is low.
By standardizing the transformation and density clustering of business data, identify outlier data and data clusters, perform attribute correlation analysis, generate attribute correlation matrix and graph, determine attribute correlation clusters, and select data with the lowest repair cost as the target data for repair .
It improves the accuracy and efficiency of data repair, can accurately identify the category information and attribute correlations between business data, and ensures the accuracy and efficiency of data repair.
Smart Images

Figure CN119025328A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of big data management, and in particular to a method and device for data repair. Background Art
[0002] With the rapid development of various industries, the amount of business data in each industry is gradually increasing, and the attributes contained in the business data are gradually increasing. Taking the logistics industry as an example, with the gradual promotion and optimization of logistics services and the gradual increase in new users, the logistics data obtained by the logistics system is also becoming more and more. However, business data is usually entered manually or generated by machines. Due to human errors or machine errors, there are abnormal data in the business data. In order to ensure accurate and efficient data analysis of business data, abnormal data in business data is currently repaired. Commonly used data repair methods include: methods for repairing time series data based on time sequence relationships, and methods for repairing structured data based on attribute constraint relationships.
[0003] In the process of implementing the present invention, the inventors found that the prior art has at least the following problems:
[0004] The data application scope of the existing technology is relatively small, resulting in a large gap between the repaired business data and the real data, and the repair result is inaccurate when the time sequence relationship of the business data is not obvious or the attribute constraint relationship is not obvious. Summary of the invention
[0005] In view of this, an embodiment of the present invention provides a method and device for data repair, which can improve the efficiency and accuracy of data repair.
[0006] To achieve the above object, according to a first aspect of an embodiment of the present invention, a method for data repair is provided, comprising:
[0007] In response to receiving the business data, performing standardization conversion on the business data to obtain standardized data, and performing density clustering on the standardized data to obtain outlier data and multiple data clusters;
[0008] Perform correlation analysis on the standard attribute values included in each data cluster to obtain the corresponding attribute correlation matrix;
[0009] Convert each attribute correlation matrix into an attribute correlation graph, and obtain corresponding attribute correlation groups according to the attribute correlation graph;
[0010] According to the attribute-related group corresponding to each data cluster, target data is determined from the multiple data clusters, and the business data corresponding to the outlier data is updated according to the business data corresponding to the target data.
[0011] Optionally, correlation analysis is performed on the standard attribute values included in each data cluster to obtain a corresponding attribute correlation matrix, including:
[0012] According to a preset order of arrangement of the attribute groups, sequentially determining a first attribute identifier and a second attribute identifier included in each attribute group;
[0013] In each data cluster, determining the attribute-related value corresponding to each attribute group according to the mean difference of all standard attribute values having the first attribute identifier and the mean difference of all standard attribute values having the second attribute identifier;
[0014] The attribute correlation matrix corresponding to each data cluster is generated according to the attribute correlation value corresponding to each attribute group in each data cluster.
[0015] Optionally, generating an attribute correlation matrix corresponding to each data cluster according to the attribute correlation value corresponding to each attribute group in each data cluster includes:
[0016] Determine the relevance level of the attribute relevance values corresponding to each attribute group in each data cluster;
[0017] Determine whether the attribute-related value in each data cluster meets the screening condition according to the correlation level, and use the attribute-related value that meets the screening condition as the first target attribute-related value;
[0018] An attribute correlation matrix corresponding to each data cluster is generated according to the first target attribute correlation value in each data cluster.
[0019] Optionally, each attribute correlation matrix is converted into an attribute correlation graph, including:
[0020] Initialize the attribute correlation graph corresponding to each attribute correlation matrix;
[0021] The attribute identifiers included in each attribute correlation matrix are used as vertices of the corresponding attribute correlation graph;
[0022] Acquire the attribute correlation values included in each attribute correlation matrix, and determine the second target attribute correlation value that meets the preset correlation condition;
[0023] Determine a plurality of target attribute identifiers corresponding to the second target attribute related value;
[0024] In the attribute correlation graph corresponding to each attribute correlation matrix, edges are generated between vertices corresponding to the multiple target attribute identifiers.
[0025] Optionally, obtaining a corresponding attribute correlation group according to the attribute correlation graph includes:
[0026] Traversing the attribute correlation graph, and splitting the attribute correlation graph into one or more connected components according to the vertices and edges included in the attribute correlation graph;
[0027] Determine the target vertices included in each connected component that do not meet the preset pruning rules;
[0028] The target vertex and the edge associated with the target vertex included in each connected component are deleted, and the connected component after the deletion is used as the attribute-related group corresponding to the attribute-related graph.
[0029] Optionally, determining target data from the multiple data clusters according to the attribute-related group corresponding to each data cluster includes:
[0030] According to the preset traversal order, the attribute-related groups corresponding to each data cluster are determined in turn;
[0031] Determine the repair cost corresponding to each standardized data according to the standardized data included in each data cluster, the corresponding attribute-related group and the outlier data;
[0032] The standardized data with the minimum repair cost is used as the target data.
[0033] Optionally, updating the business data corresponding to the outlier data according to the business data corresponding to the target data includes:
[0034] Determine a target data cluster corresponding to the target data, and determine a target attribute related group corresponding to the target data cluster;
[0035] Determine the target attribute name included in the target attribute related group;
[0036] The business data corresponding to the outlier data is taken as the first business data, the business data corresponding to the target data is taken as the second business data, and the original attribute value corresponding to the target attribute name in the first business data is replaced with the original attribute value corresponding to the target attribute name in the second business data.
[0037] According to a second aspect of an embodiment of the present invention, there is provided a data repair device, comprising:
[0038] A clustering module, configured to, in response to receiving the business data, perform standardization conversion on the business data to obtain standardized data, and perform density clustering on the standardized data to obtain outlier data and multiple data clusters;
[0039] An analysis module is used to perform correlation analysis on the standard attribute values included in each data cluster to obtain a corresponding attribute correlation matrix;
[0040] A conversion module, used for converting each attribute correlation matrix into an attribute correlation graph, and obtaining a corresponding attribute correlation group according to the attribute correlation graph;
[0041] The updating module is used to determine the target data from the multiple data clusters according to the attribute-related group corresponding to each data cluster, and update the business data corresponding to the outlier data according to the business data corresponding to the target data.
[0042] Optionally, correlation analysis is performed on the standard attribute values included in each data cluster to obtain a corresponding attribute correlation matrix, including:
[0043] According to a preset order of arrangement of the attribute groups, sequentially determining a first attribute identifier and a second attribute identifier included in each attribute group;
[0044] In each data cluster, determining the attribute-related value corresponding to each attribute group according to the mean difference of all standard attribute values having the first attribute identifier and the mean difference of all standard attribute values having the second attribute identifier;
[0045] The attribute correlation matrix corresponding to each data cluster is generated according to the attribute correlation value corresponding to each attribute group in each data cluster.
[0046] Optionally, generating an attribute correlation matrix corresponding to each data cluster according to the attribute correlation value corresponding to each attribute group in each data cluster includes:
[0047] Determine the relevance level of the attribute relevance values corresponding to each attribute group in each data cluster;
[0048] Determine whether the attribute-related value in each data cluster meets the screening condition according to the correlation level, and use the attribute-related value that meets the screening condition as the first target attribute-related value;
[0049] An attribute correlation matrix corresponding to each data cluster is generated according to the first target attribute correlation value in each data cluster.
[0050] Optionally, each attribute correlation matrix is converted into an attribute correlation graph, including:
[0051] Initialize the attribute correlation graph corresponding to each attribute correlation matrix;
[0052] The attribute identifiers included in each attribute correlation matrix are used as vertices of the corresponding attribute correlation graph;
[0053] Acquire the attribute correlation values included in each attribute correlation matrix, and determine the second target attribute correlation value that meets the preset correlation condition;
[0054] Determine a plurality of target attribute identifiers corresponding to the second target attribute related value;
[0055] In the attribute correlation graph corresponding to each attribute correlation matrix, edges are generated between vertices corresponding to the multiple target attribute identifiers.
[0056] Optionally, obtaining a corresponding attribute correlation group according to the attribute correlation graph includes:
[0057] Traversing the attribute correlation graph, and splitting the attribute correlation graph into one or more connected components according to the vertices and edges included in the attribute correlation graph;
[0058] Determine the target vertices included in each connected component that do not meet the preset pruning rules;
[0059] The target vertex and the edge associated with the target vertex included in each connected component are deleted, and the connected component after the deletion is used as the attribute-related group corresponding to the attribute-related graph.
[0060] Optionally, determining target data from the multiple data clusters according to the attribute-related group corresponding to each data cluster includes:
[0061] According to the preset traversal order, the attribute-related groups corresponding to each data cluster are determined in turn;
[0062] Determine the repair cost corresponding to each standardized data according to the standardized data included in each data cluster, the corresponding attribute-related group and the outlier data;
[0063] The standardized data with the minimum repair cost is used as the target data.
[0064] Optionally, updating the business data corresponding to the outlier data according to the business data corresponding to the target data includes:
[0065] Determine a target data cluster corresponding to the target data, and determine a target attribute related group corresponding to the target data cluster;
[0066] Determine the target attribute name included in the target attribute related group;
[0067] The business data corresponding to the outlier data is taken as the first business data, the business data corresponding to the target data is taken as the second business data, and the original attribute value corresponding to the target attribute name in the first business data is replaced with the original attribute value corresponding to the target attribute name in the second business data.
[0068] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:
[0069] one or more processors;
[0070] a storage device for storing one or more programs,
[0071] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any one of the above embodiments.
[0072] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described in any one of the above embodiments is implemented.
[0073] An embodiment of the above invention has the following advantages or beneficial effects: by density clustering the business data, outlier data that needs to be repaired can be preliminarily identified, and other business data before the outlier data can be classified, so as to quickly and efficiently determine the category information between the business data; by performing correlation analysis on each data cluster to obtain a correlation matrix, and generating attribute correlation groups according to the correlation matrix, the correlation between the attributes included in the business data can be accurately and effectively determined, so as to facilitate subsequent data repair based on correlation analysis; the target data is determined according to the attribute correlation group, and the original attribute values of the outlier data are repaired according to the original attribute values of the target data, so as to improve the efficiency and accuracy of data repair; the attribute correlation values of the first attribute identifier and the second attribute identifier are determined according to the mean difference of the two attribute identifiers, so as to accurately quantify the correlation between the first attribute identifier and the second attribute identifier, and then obtain an accurate attribute correlation matrix; grading the attribute correlation values can improve the flexibility of generating the attribute correlation matrix; setting the correlation condition can improve the flexibility of generating the attribute correlation graph; deleting the target vertex and the corresponding edge in the attribute correlation group can further improve the accuracy of the attribute correlation group; selecting the standardized data with the smallest repair cost as the target data can further improve the efficiency and accuracy of data repair.
[0074] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS
[0075] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.
[0076] Figure 1 is a schematic diagram of the main process of the data repair method according to an embodiment of the present invention;
[0077] Figure 2 is a schematic diagram of an attribute correlation graph and a corresponding attribute correlation group of a method for data repair according to a reference embodiment of the present invention;
[0078] Figure 3is a schematic diagram of the overall process of a data repair method according to a reference embodiment of the present invention;
[0079] Figure 4 is a schematic diagram of business data conversion according to a method for data repairing that can be used as a reference embodiment of the present invention;
[0080] Figure 5 is a schematic diagram of the main process of a data repair method according to a reference embodiment of the present invention;
[0081] Figure 6 is a schematic diagram of the main process of a data repair method according to another reference embodiment of the present invention;
[0082] Figure 7 is a schematic diagram of the main process of a data repair method according to yet another reference embodiment of the present invention;
[0083] Figure 8 is a schematic diagram of main modules of a data repair device according to an embodiment of the present invention;
[0084] Fig. 9 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;
[0085] Fig.10 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION
[0086] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0087] It should be noted that in the technical solution of the present invention, the collection, use, storage, sharing and transfer of user personal information involved are in compliance with the provisions of relevant laws and regulations, and it is necessary to inform the user and obtain the user's consent or authorization. When applicable, the user's personal information is de-identified and / or anonymized and / or encrypted.
[0088] With the rapid development of various industries, the amount of business data in each industry is gradually increasing, and the attributes contained in the business data are gradually increasing. Taking the logistics industry as an example, with the gradual promotion and optimization of logistics services and the gradual increase in new users, the logistics data obtained by the logistics system is also becoming more and more. However, business data is usually entered manually or generated by machines. Due to human errors (for example, typos, missing decimal points, etc.) or machine errors (for example, system errors of sensors, data loss during data transmission, etc.), there are abnormal data in the business data. In order to ensure accurate and efficient data analysis of business data, abnormal data in business data is currently repaired. Commonly used data repair methods include: methods for repairing time series data based on time sequence relationships, and methods for repairing structured data based on attribute constraint relationships.
[0089] The data application scope of the existing technology is relatively small, resulting in a large gap between the repaired business data and the real data when the time sequence relationship of the business data is not obvious or the attribute constraint relationship is not obvious, the repair result is inaccurate and the repair efficiency is low.
[0090] In view of this, according to a first aspect of an embodiment of the present invention, a method for data repair is provided.
[0091] Figure 1 FIG. 1 is a schematic diagram of the main process of the method for data repair according to an embodiment of the present invention. Figure 1 As shown, the data repair method according to the embodiment of the present invention mainly includes the following steps S101 to S103.
[0092] Step S101, in response to receiving business data, performing standardization conversion on the business data to obtain standardized data, and performing density clustering on the standardized data to obtain outlier data and multiple data clusters.
[0093] The method described in the embodiment of the present invention is used to repair business data in multiple industries. Therefore, the business data received by the execution subject of the embodiment of the present invention can be business data of various industries, such as logistics data, e-commerce data, medical data, banking business data, insurance business data, etc. The received business data includes multiple attributes, for example, logistics data includes: cargo type, cargo size, cargo weight, transportation time, transportation origin, transportation destination, etc., e-commerce data includes: commodity type, commodity price, discount information, purchase quantity, payment method, etc., medical data includes: disease name, department name, doctor name, doctor title, consultation time, etc.
[0094] After receiving the business data, each attribute of the business data is standardized respectively, and the business data is converted into a numerical value. Since there are multiple business data, multiple numerical values corresponding to the same attribute are determined, and the statistical values of the multiple numerical values are calculated, and each numerical value corresponding to the attribute is standardized according to the statistical value; for example, the average value and standard deviation of the multiple numerical values are calculated, and then each numerical value is subtracted from the average value, and the subtraction result is divided by the standard deviation, and the division result is used as the standardized numerical value of each numerical value corresponding to the attribute; for another example, the maximum value and the minimum value of the multiple numerical values are determined, the first difference between each numerical value and the minimum value is determined, the second difference between the maximum value and the minimum value is determined, the first difference is divided by the second difference, and the division result is used as the standardized numerical value of each numerical value corresponding to the attribute. Each attribute of the business data is standardized so that the values of different attributes after standardization have the same order of magnitude and value range.
[0095] Exemplarily, the received business data is e-commerce data, including: e-commerce data A1 "{'product type':'clothing';'product price':'100'}", e-commerce data A2 "{'product type':'furniture';'product price':'500'}", e-commerce data A3 "{'product type':'electrical appliances';'product price':'800'}", and according to a pre-set data conversion table, "clothing" is converted to "1", "furniture" is converted to "10", and "electrical appliances" is converted to "15". Furniture is more similar to electrical appliances, so the converted data is closer; the e-commerce data A1, e-commerce data A2 and e-commerce data A3 converted into numerical values are standardized using related methods such as "StandardScaler()" to obtain standardized data.
[0096] After obtaining the standardized data, density distance is performed on the standardized data according to the preset neighborhood distance and neighborhood density. The neighborhood distance represents the maximum distance between data of the same type, and the neighborhood density represents the minimum number of data of the same type. Specifically, the distance between different standardized data is calculated, for example, by Euclidean distance calculation; when the distance between two standardized data is less than or equal to the neighborhood distance, and the number of standardized data within the neighborhood distance is greater than or equal to the neighborhood density, the standardized data within the neighborhood distance is determined to be of the same type, and the same type of data is regarded as a data cluster. After clustering all standardized data, the standardized data that does not belong to any data cluster is outlier data, and outlier data is data that needs to be repaired. Density clustering algorithms such as DBSCAN can be used to perform density clustering on standardized data.
[0097] Exemplarily, the preset neighborhood distance is 2, the neighborhood density is 2, the distance between the standardized data B1 and the standardized data B2 is 2, the distance between the standardized data B1 and the standardized data B3 is 5, and the distance between the standardized data B2 and the standardized data B3 is 6; therefore, the standardized data B1 and the standardized data B2 are regarded as a data cluster, and the standardized data B3 is regarded as outlier data.
[0098] Standardizing business data can convert multiple attributes included in the business data into numerical values, making it easier to determine the correlation between different attributes later; clustering standardized data can accurately determine the data that needs to be repaired and preliminarily determine the correlation between the data.
[0099] Step S102: performing correlation analysis on the standard attribute values included in each data cluster to obtain a corresponding attribute correlation matrix.
[0100] Each data cluster includes one or more standardized data, each standardized data includes multiple attributes, each attribute corresponds to an attribute identifier and a standard attribute value, and the attribute identifiers included in different standardized data correspond to the same. In each data cluster, the standard attribute values corresponding to the same attribute identifier included in all standardized data are obtained to obtain the standard attribute value set corresponding to the attribute identifier. A correlation analysis is performed on the standard attribute value set corresponding to each two different attribute identifiers, for example, the mean, variance, standard deviation, median, mode and other statistical data of each standard attribute value set are calculated, and then the statistical data are subtracted or divided to obtain the attribute correlation values between different attribute identifiers, and an attribute correlation value corresponding to each two attribute identifiers is constructed according to the obtained attribute correlation values. The number of rows and columns of the attribute correlation matrix are equal to the number of attribute identifiers. The larger the attribute correlation value corresponding to the two attribute identifiers, the higher the correlation between the two attribute identifiers.
[0101] Exemplarily, the average value of the standard attribute value set corresponding to attribute identifier C1 is 2, the average value of the standard attribute value set corresponding to attribute identifier C2 is 3, and the average value of the standard attribute value set corresponding to attribute identifier C3 is 5; the calculation formula for the attribute correlation value is "1 / |avg1-avg2|", where "avg1" represents the average value corresponding to the first attribute identifier, and "avg2" represents the average value corresponding to the second attribute identifier. From the above formula, it can be obtained that the attribute correlation value between attribute identifier C1 and attribute identifier C2 is: 1 / |2-3|=1, the attribute correlation value between attribute identifier C2 and attribute identifier C3 is: 1 / |3-5|=0.5, and the attribute correlation value between attribute identifier C1 and attribute identifier C3 is: 1 / |2-5|=0.33.
[0102] Based on the correlation analysis of the standardized data in the data cluster, the attribute correlation matrix is obtained, which can reflect the degree of correlation between different attributes in the data cluster, and facilitate the subsequent data repair of outlier data based on the correlation between attributes.
[0103] According to a reference embodiment of the present invention, when performing correlation analysis, the first attribute identifier and the second attribute identifier included in each attribute group are determined in sequence according to the preset attribute group arrangement order. The attribute group is a combination of the first attribute identifier and the second attribute identifier, and different attribute groups have different attribute identifier combinations. In each data cluster, the attribute correlation value corresponding to each attribute group is determined according to the mean difference of all standard attribute values with the first attribute identifier and the mean difference of all standard attribute values with the second attribute identifier; specifically, the first average value is determined according to all standard attribute values corresponding to the first attribute identifier, each standard attribute value with the first attribute identifier is subtracted from the first average value to obtain the corresponding first average difference, the second average value is determined according to all standard attribute values corresponding to the second attribute identifier, each standard attribute value with the second attribute identifier is subtracted from the second average value to obtain the corresponding second average difference, the first average difference and the second average difference of the same standardized data are multiplied to obtain the average difference product, and all the average difference products included in each attribute group in the data cluster are summed to obtain the attribute correlation value corresponding to each attribute group. According to the arrangement order of the attribute identifiers in the standardized data, the attribute correlation values corresponding to each attribute group in each data cluster are arranged to generate the attribute correlation matrix corresponding to each data cluster.
[0104] For example, in the kth data cluster, there are p standardized data, each of which has q attributes; the attribute correlation value r between the i-th attribute and the j-th attribute is preset. ij The calculation formula is: Among them, u x represents the xth standardized data in the data cluster, S i represents the i-th attribute, S i (u x ) represents the standard attribute value of the i-th attribute of the x-th standardized data, S j (u x ) represents the standard attribute value of the jth attribute of the xth standardized data, represents the average value of all standard attribute values corresponding to the i-th attribute, The calculation formula is: represents the average value of all standard attribute values corresponding to the jth attribute, The calculation formula is: The numerical range of each attribute correlation value is between -1 and 1. The attribute correlation value between any two attributes is determined according to the above attribute correlation value calculation formula. According to the order of attribute identifiers from small to large, the attribute correlation matrix corresponding to the kth data cluster is generated as follows: Among them, r ij The value is equal to r ji The value of .
[0105] By determining the attribute correlation values of the two attribute identifiers according to the mean difference of the first attribute identifier and the mean difference of the second attribute identifier, the correlation between the first attribute identifier and the second attribute identifier can be accurately quantified, thereby obtaining an accurate attribute correlation matrix.
[0106] According to another reference embodiment of the present invention, after obtaining the attribute correlation value corresponding to each attribute group in each data cluster, the correlation level of the attribute correlation value corresponding to each attribute group in each data cluster is determined, the correlation level indicates the strength of the correlation between the two attributes corresponding to the attribute correlation value, the correlation level includes multiple levels, and the levels corresponding to attribute correlation values falling in different numerical ranges are different; one or more levels are pre-selected as filtering conditions, and whether the attribute correlation values in each data cluster meet the filtering conditions is judged according to the correlation level, that is, whether the attribute correlation values in each data cluster belong to the level included in the filtering conditions is judged, and when the attribute correlation value belongs to the level included in the filtering conditions, the attribute correlation value is judged to meet the filtering conditions, and the attribute correlation value that meets the filtering conditions is used as the first target attribute correlation value; and the attribute correlation matrix corresponding to each data cluster is generated according to the first target attribute correlation value in each data cluster.
[0107] Exemplarily, the preset relevance levels are shown in the following table:
[0108] <![CDATA[|r ij |]]> [0-0.4] (0.4-0.6] (0.6-0.8] (0.8-1] Relevance level Weak correlation Moderately related Strong correlation Very strong correlation
[0109] Among them, |r ij| represents the absolute value of the attribute correlation value between the i-th attribute and the j-th attribute. When the absolute value of the attribute correlation value is greater than or equal to 0 and less than or equal to 0.4, the correlation level between the i-th attribute and the j-th attribute is determined to be weakly correlated. When the absolute value of the attribute correlation value is greater than 0.4 and less than or equal to 0.6, the correlation level between the i-th attribute and the j-th attribute is determined to be moderately correlated. When the absolute value of the attribute correlation value is greater than 0.6 and less than or equal to 0.8, the correlation level between the i-th attribute and the j-th attribute is determined to be strongly correlated. When the absolute value of the attribute correlation value is greater than 0.8 and less than or equal to 1, the correlation level between the i-th attribute and the j-th attribute is determined to be extremely strongly correlated. Attribute correlation values with correlation levels of moderate correlation, strong correlation or extremely strong correlation are determined as attribute correlation values that meet the screening conditions. A corresponding attribute correlation matrix is generated according to the attribute correlation values that meet the screening conditions. The attribute correlation values that do not meet the screening conditions in the attribute correlation matrix are set to 0.
[0110] Setting the correlation level can pay more attention to the correlation between the attributes that meet the screening conditions, improve the flexibility of the attribute correlation matrix, and simplify the attribute correlation matrix.
[0111] Step S103: convert each attribute correlation matrix into an attribute correlation graph, and obtain corresponding attribute correlation groups according to the attribute correlation graph.
[0112] After obtaining the attribute correlation matrix, the attribute correlation graph corresponding to the attribute correlation matrix is initialized, and each attribute included in the attribute correlation matrix is used as a vertex of the attribute correlation graph. It is determined whether it is necessary to generate an edge corresponding to the attribute correlation graph according to the attribute correlation value; for example, each attribute correlation value is rounded according to a preset number of decimal places, and it is determined whether the attribute correlation value after rounding is equal to zero. If the attribute correlation value after rounding is not equal to zero, it is determined that the corresponding edge needs to be generated; for another example, when the attribute correlation value is set to 0 according to the correlation level, it is directly determined that the attribute correlation value that is not 0 needs to generate a corresponding edge; if it is determined that an edge needs to be generated, the two target attributes corresponding to the attribute correlation value are determined, and an edge is generated between the vertices corresponding to the above two target attributes in the attribute correlation graph. After all vertices are generated and the corresponding edges are generated according to the determination results, the attribute correlation graph corresponding to the attribute correlation matrix is obtained.
[0113] Exemplarily, the attribute correlation matrix includes attribute D1, attribute D2 and attribute D3, wherein the attribute correlation value d12 between attribute D1 and attribute D2 is 0.56, the attribute correlation value d23 between attribute D2 and attribute D3 is 0.34, and the attribute correlation value d13 between attribute D1 and attribute D3 is 0.04. One decimal place is retained for each attribute correlation value, and d12 is equal to 0.6, d23 is equal to 0.3, and d13 is equal to 0; then the attribute correlation graph is initialized, and attribute D1, attribute D2, and attribute D3 are respectively used as vertices, an edge is generated between the vertices corresponding to attribute D1 and attribute D2, and an edge is generated between attribute D2 and attribute D3, to obtain the final attribute correlation graph.
[0114] After obtaining the attribute correlation graph, the attribute correlation graph is split into multiple attribute correlation groups. The attribute correlation group represents a subgraph of the attribute correlation graph. The vertices and edges included in the attribute correlation group are all from the attribute correlation graph. Different attribute correlation groups include different vertices and edges. Each attribute correlation group includes one or more vertices. There is at least one path consisting of one or more edges between any two vertices included in each attribute correlation group. The attribute correlation graph is split according to the above rules to obtain the attribute correlation group corresponding to the attribute correlation graph.
[0115] Exemplarily, the attribute correlation graph includes: vertex E1, vertex E2, and vertex E3, wherein there is an edge between vertex E1 and vertex E2, and there is no edge between vertex E3 and vertex E1 or vertex E2. Then, vertex E1 and vertex E2 and the edge between vertex E1 and vertex E2 are regarded as an attribute correlation group, and vertex E3 is regarded as an attribute correlation group alone.
[0116] Splitting the attribute correlation graph into multiple attribute correlation groups can further strengthen the correlation between attributes and improve the accuracy of data repair.
[0117] According to a reference embodiment of the present invention, when converting each attribute correlation matrix into an attribute correlation graph, the attribute correlation graph corresponding to each attribute correlation matrix is initialized; the attribute identifier included in each attribute correlation matrix is used as the vertex of the corresponding attribute correlation graph, and the attribute identifier represents the name of the attribute or the number, code, etc. of the attribute. The attribute correlation value included in each attribute correlation matrix is obtained, and the second target attribute correlation value that meets the preset correlation condition is determined, and the correlation condition is used to screen the second target attribute correlation value; for example, the correlation condition is to determine whether the attribute correlation value is greater than or equal to the preset correlation threshold. When the attribute correlation value is greater than or equal to the correlation threshold, it is determined that the attribute correlation value meets the correlation condition, and then the edge between the vertices corresponding to the two attribute identifiers included in the attribute correlation value is generated. When the attribute correlation value is less than the correlation threshold, it is determined that the attribute correlation value does not meet the correlation condition, and the edge between the vertices corresponding to the two attribute identifiers included in the attribute correlation value is not generated. Each attribute correlation value is judged according to the correlation condition, so as to generate an attribute correlation graph.
[0118] Figure 2 FIG. 1 is a schematic diagram of an attribute correlation graph and a corresponding attribute correlation group of a method for data repair according to a reference embodiment of the present invention. Figure 2 As shown in the figure, the attribute correlation matrix is a 5×5 matrix, indicating that each standardized data includes 5 attributes. Assume that the attribute correlation value of the attribute itself is 1, that is, the diagonal values of the attribute correlation matrix are all 1. Each attribute is taken as a vertex of the attribute correlation graph, and the attribute correlation graph has 5 vertices, namely, attribute 1, attribute 2, attribute 3, attribute 4, and attribute 5. When the attribute correlation value is greater than or equal to 0.5, it is determined that the attribute correlation value meets the correlation condition, and then the corresponding edge is generated, and the edge between attribute 1 and attribute 2, the edge between attribute 1 and attribute 3, the edge between attribute 2 and attribute 3, and the edge between attribute 3 and attribute 5 are obtained, and then the complete attribute correlation graph corresponding to the attribute correlation matrix is obtained.
[0119] Determining whether the attribute correlation values included in the attribute correlation matrix meet the correlation conditions and generating the corresponding attribute correlation graph according to the determination result can improve the generation efficiency and flexibility of the attribute correlation graph and facilitate further correlation analysis based on the attribute correlation graph.
[0120] According to another reference embodiment of the present invention, when obtaining the corresponding attribute-related group according to the attribute-related graph, the attribute-related graph is traversed, for example, the attribute-related graph is traversed in depth first (DFS) or breadth first (BFS); according to the vertices and edges included in the attribute-related graph, the attribute-related graph is split into one or more connected components, and there is at least one path between any two vertices in the connected components. The target vertices included in each connected component that do not meet the preset pruning rules are determined, and the pruning rules are preset. The pruning rules include: Condition (1) For each connected component, assuming that the number of vertices included in the connected component is m, the degree of each vertex included in the connected component should be greater than or equal to m / 2; Condition (2) There is no target vertex that belongs to the attribute-related graph and does not belong to the target connected component, so that the union of the target vertex and the connected component meets condition (1). According to the above pruning rules, each vertex included in the attribute-related graph is judged, and the vertices and corresponding edges that meet the pruning rules are retained, and the vertices that do not meet the pruning rules are split from the corresponding connected components. The split connected components are used as the attribute-related group corresponding to the attribute-related graph.
[0121] For example, Figure 2 As shown, the attribute correlation graph includes two connected components, namely, connected component 1 consisting of five vertices and corresponding edges, namely, attribute 1, attribute 2, attribute 3, and attribute 5, and connected component 2 consisting of vertices of attribute 4. The two connected components are traversed in depth first. Since attribute 5 in connected component 1 does not meet the pruning rule, attribute 5 is split from connected component 1, and then attribute correlation group 201 is obtained. Among them, attribute correlation group 201 includes 3 vertices, and the degree of each vertex is greater than 3 / 2 (i.e., 1.5). There is no target attribute other than attribute correlation group 201 so that the union of the target attribute and attribute correlation group 201 meets the above requirements on degree. Therefore, the attribute correlation group corresponding to the attribute correlation graph is determined to be Figure 2 The attribute related group 201 is marked with a dotted line.
[0122] By traversing the attribute-related groups and determining the attribute-related groups according to their definitions, the correlation relationship between the attributes can be further obtained, thereby improving the efficiency and accuracy of obtaining the attribute-related groups.
[0123] Step S104 , determining target data from the multiple data clusters according to the attribute-related group corresponding to each data cluster, and updating the business data corresponding to the outlier data according to the business data corresponding to the target data.
[0124] The target data is standardized data used for data repair of outlier data. Each standardized data in multiple data clusters is judged separately from the outlier data. For example, according to the target attribute identifier included in the attribute-related group, the Euclidean distance between each standardized data and the outlier data is calculated, and the standardized data with the smallest Euclidean distance is used as the target data; for another example, according to the target attribute identifier included in the attribute-related group, the variance or standard deviation between each target attribute of each standardized data and each target attribute included in the outlier data is calculated, and the obtained variances or standard deviations are summed, and the standardized data with the smallest sum of variances or the smallest sum of standard deviations is used as the target data. Then, the target data is converted into corresponding business data, the outlier data is converted into corresponding business data, and the business data corresponding to the target data is used to replace the business data corresponding to the outlier data.
[0125] Exemplarily, data cluster F includes standardized data F1 and standardized data F2, and the attribute-related group corresponding to the data cluster includes three vertices, and the three vertices respectively represent attribute identifier g1, attribute identifier g2, and attribute identifier g3; a first variance is calculated between the attribute values corresponding to the above three attribute identifiers included in the standardized data F1 and the attribute values corresponding to the above three attribute identifiers included in the outlier data, and a second variance is calculated between the attribute values corresponding to the above three attribute identifiers included in the standardized data F2 and the attribute values corresponding to the above three attribute identifiers included in the outlier data; when the first variance is less than or equal to the second variance, the business data corresponding to the standardized data F1 is used to replace the business data corresponding to the outlier data; when the first variance is greater than the second variance, the business data corresponding to the standardized data F2 is used to replace the business data corresponding to the outlier data.
[0126] Determining target data according to attribute-related groups can improve the accuracy and efficiency of acquiring target data, and using the business data corresponding to the target data to repair the business data corresponding to the outlier data can improve the accuracy of data repair.
[0127] According to a reference embodiment of the present invention, when determining the target data, the attribute-related groups corresponding to each data cluster are determined in turn according to the preset traversal order; the attribute values in the standardized data corresponding to each data cluster are obtained according to the attribute-related groups corresponding to each data cluster; the attribute values in the outlier data are obtained according to the attribute-related groups corresponding to each data cluster; the repair cost corresponding to each standardized data is determined according to the two groups of attribute values, and the repair cost represents the subtraction of the two groups of attribute values, the sum of the squares of the differences and the square root, each attribute-related group corresponds to a repair cost, and the repair costs corresponding to all attribute-related groups corresponding to each standardized data are summed to obtain the repair cost corresponding to each standardized data. The repair costs corresponding to each standardized data are compared, and the standardized data with the smallest repair cost is used as the target data.
[0128] For example, assume that the outlier data is h 1 , standardized data is h 2 , the repair cost is S, and the calculation formula of S is: Where O represents the standardized data h 2 The attribute-related cluster set corresponding to the data cluster, cost(h 1 ,h 2 ,o u ) represents the standardized data h 2 The corresponding attribute related group o u The corresponding modification cost, o u The technical formula for repairing the u-th attribute-related group in the attribute-related group set O is: Among them, d v (h 1 ) represents the outlier data and the corresponding attribute related group o u The attribute value of the vth attribute in d v (h 2 ) represents the group o related to the normalized data pair response attribute u The attribute value of the vth attribute in the , subtract the corresponding attribute values and then calculate the square sum to get the attribute related group o u Corresponding repair costs. Compare the repair costs corresponding to each standardized data, and take the standardized data with the smallest repair cost as the target data.
[0129] Selecting standardized data with the lowest repair cost as target data can further improve data repair efficiency and accuracy.
[0130] According to another reference embodiment of the present invention, when updating the business data corresponding to the outlier data according to the business data corresponding to the target data, determine the target data cluster corresponding to the target data, determine the target attribute related group corresponding to the target data cluster, and the target attribute related group is the attribute related group with the lowest repair cost in the target data cluster. The target attribute related group includes multiple target attributes, and the target attribute name included in the target attribute related group is determined. Convert the outlier data into business data, use the business data corresponding to the outlier data as the first business data, convert the target data into business data, use the business data corresponding to the target data as the second business data, and replace the original attribute value corresponding to the target attribute name in the first business data with the original attribute value corresponding to the target attribute name in the second business data, thereby completing the data repair of the outlier data.
[0131] Exemplarily, the data cluster where the target data is located includes attribute-related group x1 and attribute-related group x2, wherein attribute-related group x1 includes attribute name y1 and attribute name y2, and attribute-related group x2 includes attribute name y3, attribute name y4, and attribute name y5; the repair cost corresponding to attribute-related group x1 is 3, and the repair cost corresponding to attribute-related group x2 is 5. Therefore, attribute-related group x1 is used as the target attribute-related group, and the target data is converted into business data z1. Business data z1 is “{'y1':'1'; 'y2':'2'; 'y3':'3'; 'y4':'4'; 'y5':'5'}”, wherein “ 'y1':'1'" indicates that the original attribute value corresponding to the attribute name "y1" is 1, and so on for other attributes. The outlier data is converted into business data z2, and the business data z2 is "{'y1':'6'; 'y2':'7'; 'y3':'8'; 'y4':'9'; 'y5':'0'}". The original attribute value corresponding to the target attribute name in the business data z2 is replaced with the original attribute value corresponding to the target attribute name in the business data z1, and the repaired outlier data is "{'y1':'1'; 'y2':'2'; 'y3':'8'; 'y4':'9'; 'y5':'0'}".
[0132] Repairing outlier data according to target attribute related groups can improve the accuracy and efficiency of outlier data repair.
[0133] Figure 3 FIG. 1 is a schematic diagram of the overall process of a method for data repair according to a reference embodiment of the present invention. Figure 3 As shown, after receiving the business data, the business data is standardized and converted to obtain standardized data; the standardized data is density clustered to obtain outlier data and multiple data clusters, wherein each data cluster includes multiple standardized data; a correlation analysis is performed on each data cluster to obtain a corresponding attribute correlation matrix; a corresponding attribute correlation graph is generated according to each attribute correlation matrix, and each attribute correlation graph is traversed and pruned to obtain a corresponding attribute correlation group; the corresponding target attribute correlation group is determined according to the data cluster where the standardized data is located, and the repair cost between each standardized data and the outlier data is determined according to the target attribute correlation group, and the standardized data with the smallest repair cost is selected as the target data; the target data is converted into the first business data, and the outlier data is also converted into the second business data, and according to the target attribute identifier included in the target attribute correlation group, the original attribute value corresponding to the target attribute identifier included in the first business data is used to replace the original attribute value corresponding to the target attribute identifier included in the second business data, thereby realizing data repair of the outlier data.
[0134] Figure 4FIG. 1 is a schematic diagram of business data conversion of a method for data repair according to a reference embodiment of the present invention. Figure 4 As shown, the received business data is logistics data 401, and the logistics data includes multiple attributes such as cargo type, cargo size, cargo quantity, cargo weight, transportation starting point, and transportation destination; after the logistics data is standardized and density clustered, it is determined that the standardized data corresponding to the logistics data numbered 4 is outlier data; after correlation analysis, attribute correlation graph generation, attribute correlation group generation, and repair cost comparison, it is determined that the standardized data corresponding to the logistics data numbered 3 is target data, and the target attribute identifiers indicated by the attribute correlation group include: "cargo size and cargo weight"; the original attribute value "XLL" corresponding to the "cargo size" in the logistics data numbered 4 is modified to the original attribute value "XXL" corresponding to the "cargo size" in the logistics data numbered 3, and the original attribute value "40kg" corresponding to the "cargo weight" in the logistics data numbered 4 is modified to the original attribute value "3kg" corresponding to the "cargo weight" in the logistics data numbered 3; thereby obtaining the repaired logistics data 402 numbered 4.
[0135] Figure 5 FIG. 1 is a schematic diagram of the main process of a data repair method according to a reference embodiment of the present invention. Figure 5 As shown, the data repair method may include:
[0136] Step S501, in response to receiving the business data, performing standardization conversion on the business data to obtain standardized data, and performing density clustering on the standardized data to obtain outlier data and multiple data clusters;
[0137] Step S502, determining the first attribute identifier and the second attribute identifier included in each attribute group in sequence according to a preset attribute group arrangement order;
[0138] Step S503, in each data cluster, according to the mean difference of all standard attribute values having the first attribute identifier and the mean difference of all standard attribute values having the second attribute identifier, determining the attribute correlation value corresponding to each attribute group;
[0139] Step S504, generating an attribute correlation matrix corresponding to each data cluster according to the attribute correlation value corresponding to each attribute group in each data cluster;
[0140] Step S505, initializing the attribute correlation graph corresponding to each attribute correlation matrix;
[0141] Step S506, taking the attribute identifiers included in each attribute correlation matrix as vertices of the corresponding attribute correlation graph;
[0142] Step S507, obtaining the attribute correlation values included in each attribute correlation matrix, and determining the second target attribute correlation value that meets the preset correlation condition;
[0143] Step S508, determining a plurality of target attribute identifiers corresponding to the second target attribute related value;
[0144] Step S509, in the attribute correlation graph corresponding to each attribute correlation matrix, edges are generated between the vertices corresponding to the multiple target attribute identifiers to obtain a final attribute correlation graph;
[0145] Step S510, obtaining corresponding attribute correlation groups according to the final attribute correlation graph;
[0146] Step S511 , determining target data from the multiple data clusters according to the attribute-related group corresponding to each data cluster, and updating the business data corresponding to the outlier data according to the business data corresponding to the target data.
[0147] The specific implementation content of the data repair method of the above-mentioned reference embodiment of the present invention has been described in detail in the data repair method described above, so the repeated content will not be described again here.
[0148] Figure 6 FIG. 1 is a schematic diagram of the main process of a data repair method according to another reference embodiment of the present invention. Figure 6 As shown, the data repair method may include:
[0149] Step S601, in response to receiving the business data, performing standardization conversion on the business data to obtain standardized data, and performing density clustering on the standardized data to obtain outlier data and multiple data clusters;
[0150] Step S602, determining the first attribute identifier and the second attribute identifier included in each attribute group in sequence according to a preset attribute group arrangement order;
[0151] Step S603, in each data cluster, according to the mean difference of all standard attribute values with the first attribute identifier and the mean difference of all standard attribute values with the second attribute identifier, determining the attribute correlation value corresponding to each attribute group;
[0152] Step S604, determining the correlation level of the attribute correlation value corresponding to each attribute group in each data cluster;
[0153] Step S605, determining whether the relevance level of the attribute relevance value meets the screening condition, if so, jump to step S606, otherwise, jump to step S604;
[0154] Step S606, taking the attribute correlation values that meet the screening conditions as the first target attribute correlation values, and generating an attribute correlation matrix corresponding to each data cluster according to the first target attribute correlation values in each data cluster;
[0155] Step S607, initializing the attribute correlation graph corresponding to each attribute correlation matrix;
[0156] Step S608, taking the attribute identifiers included in each attribute correlation matrix as vertices of the corresponding attribute correlation graph;
[0157] Step S609, obtaining the attribute correlation values included in each attribute correlation matrix, and determining the second target attribute correlation value that meets the preset correlation condition;
[0158] Step S610, determining a plurality of target attribute identifiers corresponding to the second target attribute related value;
[0159] Step S611, in the attribute correlation graph corresponding to each attribute correlation matrix, edges are generated between vertices corresponding to the multiple target attribute identifiers to obtain a final attribute correlation graph;
[0160] Step S612, obtaining corresponding attribute correlation groups according to the final attribute correlation graph;
[0161] Step S613, determining the attribute-related groups corresponding to each data cluster in turn according to a preset traversal order;
[0162] Step S614, determining the repair cost corresponding to each standardized data according to the standardized data, the corresponding attribute-related groups and the outlier data included in each data cluster;
[0163] Step S615, taking the standardized data with the minimum repair cost as the target data;
[0164] Step S616: updating the business data corresponding to the outlier data according to the business data corresponding to the target data.
[0165] The specific implementation content of the data repair method of another reference embodiment of the present invention has been described in detail in the data repair method described above, so the repeated content will not be described again here.
[0166] Figure 7 FIG. 1 is a schematic diagram of the main process of a data repair method according to another reference embodiment of the present invention. Figure 7 As shown, the data repair method may include:
[0167] Step S701, in response to receiving the business data, performing standardization conversion on the business data to obtain standardized data, and performing density clustering on the standardized data to obtain outlier data and multiple data clusters;
[0168] Step S702, determining the first attribute identifier and the second attribute identifier included in each attribute group in sequence according to a preset attribute group arrangement order;
[0169] Step S703, in each data cluster, according to the mean difference of all standard attribute values having the first attribute identifier and the mean difference of all standard attribute values having the second attribute identifier, determining the attribute-related value corresponding to each attribute group;
[0170] Step S704, generating an attribute correlation matrix corresponding to each data cluster according to the attribute correlation value corresponding to each attribute group in each data cluster;
[0171] Step S705, initializing the attribute correlation graph corresponding to each attribute correlation matrix;
[0172] Step S706, taking the attribute identifiers included in each attribute correlation matrix as vertices of the corresponding attribute correlation graph;
[0173] Step S707, obtaining the attribute correlation values included in each attribute correlation matrix, and determining the second target attribute correlation value that meets the preset correlation condition;
[0174] Step S708, determining a plurality of target attribute identifiers corresponding to the second target attribute related value;
[0175] Step S709, in the attribute correlation graph corresponding to each attribute correlation matrix, edges are generated between the vertices corresponding to the multiple target attribute identifiers to obtain a final attribute correlation graph;
[0176] Step S710, traversing the attribute correlation graph, and splitting the attribute correlation graph into one or more connected components according to the vertices and edges included in the attribute correlation graph;
[0177] Step S711, determining the target vertices included in each connected component that do not meet the preset pruning rules;
[0178] Step S712, deleting the target vertex and the edge associated with the target vertex included in each connected component, and using the deleted connected component as the attribute-related group corresponding to the attribute-related graph;
[0179] Step S713, determining target data from multiple data clusters according to the attribute-related group corresponding to each data cluster;
[0180] Step S714, determining a target data cluster corresponding to the target data, and determining a target attribute related group corresponding to the target data cluster;
[0181] Step S715, determining the target attribute name included in the target attribute related group, taking the business data corresponding to the outlier data as the first business data, and taking the business data corresponding to the target data as the second business data;
[0182] Step S716: Replace the original attribute value corresponding to the target attribute name in the first business data with the original attribute value corresponding to the target attribute name in the second business data.
[0183] The specific implementation content of the data repair method of another reference embodiment of the present invention has been described in detail in the data repair method described above, so the repeated content will not be described here.
[0184] According to a second aspect of an embodiment of the present invention, a data repair device is provided.
[0185] Figure 8 is a schematic diagram of the main modules of the data repair device according to an embodiment of the present invention, such as Figure 8 As shown, the data repair device 800 mainly includes:
[0186] The clustering module 801 is used for, in response to receiving the business data, performing standardization conversion on the business data to obtain standardized data, and performing density clustering on the standardized data to obtain outlier data and multiple data clusters;
[0187] An analysis module 802 is used to perform correlation analysis on the standard attribute values included in each data cluster to obtain a corresponding attribute correlation matrix;
[0188] A conversion module 803 is used to convert each attribute correlation matrix into an attribute correlation graph, and obtain a corresponding attribute correlation group according to the attribute correlation graph;
[0189] The updating module 804 is used to determine target data from the multiple data clusters according to the attribute-related group corresponding to each data cluster, and update the business data corresponding to the outlier data according to the business data corresponding to the target data.
[0190] According to a reference embodiment of the present invention, correlation analysis is performed on the standard attribute values included in each data cluster to obtain a corresponding attribute correlation matrix, including:
[0191] According to a preset order of arrangement of the attribute groups, sequentially determining a first attribute identifier and a second attribute identifier included in each attribute group;
[0192] In each data cluster, determining the attribute-related value corresponding to each attribute group according to the mean difference of all standard attribute values having the first attribute identifier and the mean difference of all standard attribute values having the second attribute identifier;
[0193] The attribute correlation matrix corresponding to each data cluster is generated according to the attribute correlation value corresponding to each attribute group in each data cluster.
[0194] According to another reference embodiment of the present invention, generating an attribute correlation matrix corresponding to each data cluster according to the attribute correlation value corresponding to each attribute group in each data cluster includes:
[0195] Determine the relevance level of the attribute relevance values corresponding to each attribute group in each data cluster;
[0196] Determine whether the attribute-related value in each data cluster meets the screening condition according to the correlation level, and use the attribute-related value that meets the screening condition as the first target attribute-related value;
[0197] An attribute correlation matrix corresponding to each data cluster is generated according to the first target attribute correlation value in each data cluster.
[0198] According to another reference embodiment of the present invention, each attribute correlation matrix is converted into an attribute correlation graph, including:
[0199] Initialize the attribute correlation graph corresponding to each attribute correlation matrix;
[0200] The attribute identifiers included in each attribute correlation matrix are used as vertices of the corresponding attribute correlation graph;
[0201] Acquire the attribute correlation values included in each attribute correlation matrix, and determine the second target attribute correlation value that meets the preset correlation condition;
[0202] Determine a plurality of target attribute identifiers corresponding to the second target attribute related value;
[0203] In the attribute correlation graph corresponding to each attribute correlation matrix, edges are generated between vertices corresponding to the multiple target attribute identifiers.
[0204] According to another reference embodiment of the present invention, obtaining a corresponding attribute correlation group according to the attribute correlation graph includes:
[0205] Traversing the attribute correlation graph, and splitting the attribute correlation graph into one or more connected components according to the vertices and edges included in the attribute correlation graph;
[0206] Determine the target vertices included in each connected component that do not meet the preset pruning rules;
[0207] The target vertex and the edge associated with the target vertex included in each connected component are deleted, and the connected component after the deletion is used as the attribute-related group corresponding to the attribute-related graph.
[0208] According to another reference embodiment of the present invention, determining target data from the multiple data clusters according to the attribute-related group corresponding to each data cluster includes:
[0209] According to the preset traversal order, the attribute-related groups corresponding to each data cluster are determined in turn;
[0210] Determine the repair cost corresponding to each standardized data according to the standardized data included in each data cluster, the corresponding attribute-related group and the outlier data;
[0211] The standardized data with the minimum repair cost is used as the target data.
[0212] According to a reference embodiment of the present invention, updating the business data corresponding to the outlier data according to the business data corresponding to the target data includes:
[0213] Determine a target data cluster corresponding to the target data, and determine a target attribute related group corresponding to the target data cluster;
[0214] Determine the target attribute name included in the target attribute related group;
[0215] The business data corresponding to the outlier data is taken as the first business data, the business data corresponding to the target data is taken as the second business data, and the original attribute value corresponding to the target attribute name in the first business data is replaced with the original attribute value corresponding to the target attribute name in the second business data.
[0216] It should be noted that the specific implementation content of the data repair device described in the embodiment of the present invention has been described in detail in the data repair method described above, so the repeated content will not be described again here.
[0217] According to the technical solution of the embodiment of the present invention, by density clustering the business data, the outlier data that needs to be repaired can be preliminarily identified, and the other business data before the outlier data can be classified, so as to quickly and efficiently determine the category information between the business data; by performing correlation analysis on each data cluster to obtain a correlation matrix, and generating attribute correlation groups according to the correlation matrix, the correlation between the attributes included in the business data can be accurately and effectively determined, so as to facilitate the subsequent data repair based on the correlation analysis; the target data is determined according to the attribute correlation group, and the original attribute values of the outlier data are repaired according to the original attribute values of the target data, so as to improve the efficiency and accuracy of data repair; the attribute correlation values of the first attribute identifier and the second attribute identifier are determined according to the mean difference of the two attribute identifiers, so as to accurately quantify the correlation between the first attribute identifier and the second attribute identifier, and then obtain an accurate attribute correlation matrix; grading the attribute correlation values can improve the flexibility of generating the attribute correlation matrix; setting the correlation condition can improve the flexibility of generating the attribute correlation graph; deleting the target vertex and the corresponding edge in the attribute correlation group can further improve the accuracy of the attribute correlation group; selecting the standardized data with the smallest repair cost as the target data can further improve the efficiency and accuracy of data repair.
[0218] According to a third aspect of an embodiment of the present invention, there is provided an electronic device for data repair, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the first aspect of the embodiment of the present invention.
[0219] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method provided by the first aspect of the embodiment of the present invention is implemented.
[0220] Fig. 9 An exemplary system architecture 900 is shown to which the method or apparatus for data repair according to the embodiment of the present invention may be applied.
[0221] like Fig. 9 As shown, system architecture 900 may include terminal devices 901, 902, 903, network 904 and server 905. Network 904 is used to provide a medium for communication links between terminal devices 901, 902, 903 and server 905. Network 904 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0222] The user can use the terminal devices 901, 902, 903 to interact with the server 905 through the network 904 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 901, 902, 903, such as data repair applications, data analysis applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).
[0223] The terminal devices 901 , 902 , and 903 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.
[0224] Server 905 may be a server that provides various services, such as a background management server that provides support for data repair requests sent by upstream terminal devices 901, 902, and 903 (for example only). In response to receiving the business data, the background management server may perform standardization conversion on the business data to obtain standardized data, perform density clustering on the standardized data to obtain outlier data and multiple data clusters; perform correlation analysis on the standard attribute values included in each data cluster to obtain a corresponding attribute correlation matrix; convert each attribute correlation matrix into an attribute correlation graph, and obtain corresponding attribute correlation groups according to the attribute correlation graph; determine target data from the multiple data clusters according to the attribute correlation groups corresponding to each data cluster, and update the business data corresponding to the outlier data according to the business data corresponding to the target data; and feed back the data repair situation (for example only) to the terminal device.
[0225] It should be noted that the data repair method provided in the embodiment of the present invention is generally executed by the server 905, and accordingly, the data repair device is generally set in the server 905. The data repair method provided in the embodiment of the present invention can also be executed by the terminal devices 901, 902, and 903, and accordingly, the data repair device can be set in the terminal devices 901, 902, and 903.
[0226] It should be understood that Fig. 9 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0227] Reference below Fig.10 , which shows a schematic diagram of the structure of a computer system 1000 of a terminal device suitable for implementing an embodiment of the present invention. Fig.10 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0228] like Fig.10As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the system 1000 are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.
[0229] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.
[0230] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, the above-mentioned functions defined in the system of the embodiment of the present invention are executed.
[0231] It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In an embodiment of the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0232] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0233] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, they may be described as: a processor including a clustering module, an analysis module, a conversion module, and an update module, wherein the names of these modules do not constitute a limitation on the modules themselves in certain circumstances, for example, the clustering module may also be described as a "module for density clustering of standardized data".
[0234] As another aspect, an embodiment of the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device implements the following method: in response to receiving business data, the business data is standardized and converted to obtain standardized data, and the standardized data is density clustered to obtain outlier data and multiple data clusters; correlation analysis is performed on the standard attribute values included in each data cluster to obtain a corresponding attribute correlation matrix; each attribute correlation matrix is converted into an attribute correlation graph, and a corresponding attribute correlation group is obtained according to the attribute correlation graph; according to the attribute correlation group corresponding to each data cluster, the target data is determined from the multiple data clusters, and the business data corresponding to the outlier data is updated according to the business data corresponding to the target data.
[0235] According to the technical solution of the embodiment of the present invention, by density clustering the business data, the outlier data that needs to be repaired can be preliminarily identified, and the other business data before the outlier data can be classified, so as to quickly and efficiently determine the category information between the business data; by performing correlation analysis on each data cluster to obtain a correlation matrix, and generating attribute correlation groups according to the correlation matrix, the correlation between the attributes included in the business data can be accurately and effectively determined, so as to facilitate the subsequent data repair based on the correlation analysis; the target data is determined according to the attribute correlation group, and the original attribute values of the outlier data are repaired according to the original attribute values of the target data, so as to improve the efficiency and accuracy of data repair; the attribute correlation values of the first attribute identifier and the second attribute identifier are determined according to the mean difference of the two attribute identifiers, so as to accurately quantify the correlation between the first attribute identifier and the second attribute identifier, and then obtain an accurate attribute correlation matrix; grading the attribute correlation values can improve the flexibility of generating the attribute correlation matrix; setting the correlation condition can improve the flexibility of generating the attribute correlation graph; deleting the target vertex and the corresponding edge in the attribute correlation group can further improve the accuracy of the attribute correlation group; selecting the standardized data with the smallest repair cost as the target data can further improve the efficiency and accuracy of data repair.
[0236] The above specific implementations do not constitute a limitation on the protection scope of the embodiments of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.
Claims
1. A method for data repair, characterized in that: include: In response to receiving the business data, performing standardization conversion on the business data to obtain standardized data, and performing density clustering on the standardized data to obtain outlier data and multiple data clusters; Perform correlation analysis on the standard attribute values included in each data cluster to obtain the corresponding attribute correlation matrix; Convert each attribute correlation matrix into an attribute correlation graph, and obtain corresponding attribute correlation groups according to the attribute correlation graph; According to the attribute-related group corresponding to each data cluster, target data is determined from the multiple data clusters, and the business data corresponding to the outlier data is updated according to the business data corresponding to the target data.
2. The method according to claim 1, characterized in that Perform correlation analysis on the standard attribute values included in each data cluster to obtain the corresponding attribute correlation matrix, including: According to a preset order of arrangement of the attribute groups, sequentially determining a first attribute identifier and a second attribute identifier included in each attribute group; In each data cluster, determining the attribute-related value corresponding to each attribute group according to the mean difference of all standard attribute values having the first attribute identifier and the mean difference of all standard attribute values having the second attribute identifier; The attribute correlation matrix corresponding to each data cluster is generated according to the attribute correlation value corresponding to each attribute group in each data cluster.
3. The method according to claim 2, characterized in that Generate the attribute correlation matrix corresponding to each data cluster according to the attribute correlation value corresponding to each attribute group in each data cluster, including: Determine the relevance level of the attribute relevance values corresponding to each attribute group in each data cluster; Determine whether the attribute-related value in each data cluster meets the screening condition according to the correlation level, and use the attribute-related value that meets the screening condition as the first target attribute-related value; An attribute correlation matrix corresponding to each data cluster is generated according to the first target attribute correlation value in each data cluster.
4. The method according to claim 1, characterized in that Convert each attribute correlation matrix into an attribute correlation graph, including: Initialize the attribute correlation graph corresponding to each attribute correlation matrix; The attribute identifiers included in each attribute correlation matrix are used as vertices of the corresponding attribute correlation graph; Acquire the attribute correlation values included in each attribute correlation matrix, and determine the second target attribute correlation value that meets the preset correlation condition; Determine a plurality of target attribute identifiers corresponding to the second target attribute related value; In the attribute correlation graph corresponding to each attribute correlation matrix, edges are generated between vertices corresponding to the multiple target attribute identifiers.
5. The method according to claim 1, characterized in that Obtaining a corresponding attribute correlation group according to the attribute correlation graph includes: Traversing the attribute correlation graph, and splitting the attribute correlation graph into one or more connected components according to the vertices and edges included in the attribute correlation graph; Determine the target vertices included in each connected component that do not meet the preset pruning rules; The target vertex and the edge associated with the target vertex included in each connected component are deleted, and the connected component after the deletion is used as the attribute-related group corresponding to the attribute-related graph.
6. The method according to claim 1, characterized in that Determining target data from the plurality of data clusters according to the attribute-related groups corresponding to each data cluster includes: According to the preset traversal order, the attribute-related groups corresponding to each data cluster are determined in turn; Determine the repair cost corresponding to each standardized data according to the standardized data included in each data cluster, the corresponding attribute-related group and the outlier data; The standardized data with the minimum repair cost is used as the target data.
7. The method according to claim 1, characterized in that Updating the business data corresponding to the outlier data according to the business data corresponding to the target data includes: Determine a target data cluster corresponding to the target data, and determine a target attribute related group corresponding to the target data cluster; Determine the target attribute name included in the target attribute related group; The business data corresponding to the outlier data is taken as the first business data, the business data corresponding to the target data is taken as the second business data, and the original attribute value corresponding to the target attribute name in the first business data is replaced with the original attribute value corresponding to the target attribute name in the second business data.
8. A data repair device, characterized in that: include: A clustering module, configured to, in response to receiving the business data, perform standardization conversion on the business data to obtain standardized data, and perform density clustering on the standardized data to obtain outlier data and multiple data clusters; An analysis module is used to perform correlation analysis on the standard attribute values included in each data cluster to obtain a corresponding attribute correlation matrix; A conversion module, used for converting each attribute correlation matrix into an attribute correlation graph, and obtaining a corresponding attribute correlation group according to the attribute correlation graph; The updating module is used to determine the target data from the multiple data clusters according to the attribute-related group corresponding to each data cluster, and update the business data corresponding to the outlier data according to the business data corresponding to the target data.
9. An electronic device, characterized in that: include: One or more processors; A storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.
10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
