Method and system for detecting abnormal data in data set and data cleaning method

Abstract

The invention discloses a method for detecting abnormal data in a data set. The method comprises the following steps: acquiring a to-be-detected data set; carrying out sparse processing on the data set to construct a sparse graph; sampling and mapping the sparse graph to generate a feature point set; calculating corresponding structure entropy information according to the feature point set and generating a data space coding tree; calculating the structure entropy of leaf nodes of the data space coding tree to obtain the score of each data point in the feature point set, and obtaining the score of each data point in the to-be-detected data set according to the score and the mapping relation; and completing the detection of the abnormal data in the to-be-detected data set according to the score of each data point in the to-be-detected data set. The invention also discloses a system for realizing the method for detecting the abnormal data in the data set, and a data cleaning method comprising the method for detecting the abnormal data in the data set. According to the method, detection and corresponding cleaning of the abnormal data in the data set are achieved, the reliability is higher, and the accuracy is better.

Description

Technical Field

[0001] The present invention belongs to the field of data processing, and in particular relates to a method and system for detecting abnormal data in a data set, and a data cleaning method. Background Art

[0002] With the development of economy and technology and the improvement of people's living standards, machine learning technology has been widely applied in people's production and daily life, bringing endless convenience to people's production and life. In the process of machine learning, it is necessary to deal with anomalies in the learning dataset. Therefore, detecting anomalies in the dataset is particularly important.

[0003] Currently, common outlier detection schemes for a given dataset include proximity-based and statistical-based approaches. Proximity-based approaches use information about local density and average distance to distinguish outliers from inliers. These approaches are based on the assumption that outliers are often isolated, with lower local density and larger average distance than normal points. Proximity-based approaches often leverage local density and average distance to distinguish outliers from normal points, treating data points with lower density and their neighborhood as outliers. However, this assumption makes them difficult to distinguish between outliers and normal points with complex patterns and relationships. Statistical-based approaches, on the other hand, fit a probability distribution to the given data points and identify outliers based on how well the data fit the constructed model. However, statistical-based approaches often rely on strong assumptions that the data satisfy a certain underlying joint probability distribution and are unable to represent complex multivariate distributions. Consequently, these approaches struggle with datasets with complex patterns and distributions. Summary of the Invention

[0004] One of the objectives of the present invention is to provide a method for detecting abnormal data in a data set with high reliability and good accuracy.

[0005] A second object of the present invention is to provide a system for implementing the method for detecting abnormal data in the data set.

[0006] A third object of the present invention is to provide a data cleaning method including a method for detecting abnormal data in the data set.

[0007] The method for detecting abnormal data in the data set provided by the present invention comprises the following steps:

[0008] S1. Obtain the data set to be tested;

[0009] S2. performing sparse processing on the data set obtained in step S1 to construct a sparse graph;

[0010] S3. Sampling and mapping the sparse graph obtained in step S2 to generate a set of feature points;

[0011] S4. Calculate the corresponding structural entropy information based on the feature point set obtained in step S3 and generate a data space coding tree;

[0012] S5. Calculate the structural entropy of the leaf nodes of the data space coding tree obtained in step S4 to obtain the score of each data point in the feature point set, and obtain the score of each data point in the data set to be detected based on the score and mapping relationship of each data point;

[0013] S6. Complete the detection of abnormal data in the data set to be detected based on the score of each data point in the data set to be detected obtained in step S5.

[0014] Step S2, in which the data set obtained in step S1 is subjected to sparse processing to construct a sparse graph, specifically includes the following steps:

[0015] Initialize the core point set to a randomly selected point and iteratively construct the core point set: calculate each data point x i The distance to all core points in the core point set, and the distance to the nearest core point is set as the data point x i The distance to the core point set is to add the data point farthest from the current core point set to the core point set; repeat this step until the distance from all data points to the nearest core point is less than the set value ε, and the construction of the core point set is completed;

[0016] Calculate the median of the distance between all data points and the closest core point, and set the median as the distance threshold ε1;

[0017] Calculate for each data point x i The distance to other data points in the data set, setting the neighbor parameter N, gets the distance x from each data point i The set of the most recent N data points {x i,1 ,x i,2 ,...,x i,N}, set the point in the set to be the data point x i Neighbor points;

[0018] Filter each data point x according to the set distance threshold ε1 i Only the edges with a distance less than the threshold ε1 to its neighboring points are retained;

[0019] Finally, a sparse graph is obtained.

[0020] Step S3 of sampling and mapping the sparse graph obtained in step S2 to generate a feature point set specifically includes the following steps:

[0021] According to the sparse graph obtained in step S2, all points in the sparse graph are initialized to the point set X = {x1, x2, ..., x n}, and as the current point set, n is the total number of points;

[0022] The mapping σ of the initialization point set is for each data point x i Map to itself;

[0023] Repeat steps A to E until the set conditions are met:

[0024] A. From the current point set X i In the process, random sampling with replacement is performed to obtain a point set S of size αt i , where α is the set sampling parameter and t is the set sample cardinality parameter;

[0025] B. Calculate each point in the current point set to point set S i The distance of each point in the point set S and record the distance of each point in the current point set in the point set S i The closest point in the middle;

[0026] C. According to the current point set X i , point set S i And the set proportional parameter β, the minimum radius ρ is calculated using the following formula i :

[0027] |B(S i ,X i ,ρ i )|≥β|X i |

[0028] Where B(S i ,X i ,ρ i ) represents the current point set X i Each point and point set S i The distance between the nearest points is less than ρ i The set of points is represented as point set C i ;|B(S i ,X i ,ρ i )| represents the set B(S i ,X i ,ρ i ) the number of midpoints; |X i | represents the set X i the number of midpoints;

[0029] D. According to point set C i , the point set C i Each point in is mapped to the point set Si The closest point in the middle;

[0030] E. In point set S i Remove point set C from i The points in the , and the obtained point set is used as the current point set X i ;

[0031] The setting condition is: current point set X i The number of points in is less than the set value, and the current corresponding number of iterations r is saved;

[0032] Finally, the final current point set X r Each point in is mapped to itself;

[0033] For the current point set X r And the point set S in each iteration i For all elements x in , calculate the number of times x is mapped and use it as the weight ω of element x x ;(x,ω x ) Add to the weight set Q;

[0034] Finally, the weight set Q is set as the feature point set;

[0035] Step S4 calculates the corresponding structural entropy information based on the feature point set obtained in step S3 and generates a data space coding tree, which specifically includes the following steps:

[0036] Based on the weight set Q obtained in step S3 and the sparse graph G = (V, E, W) obtained in step S2, a new graph G' = (Q, E', W') is constructed using the following rules:

[0037] Rule 1: Only keep the data points in the data point set V that exist in the weight set Q;

[0038] Rule 2: Only retain the edges (u, v) in the edge set E that satisfy u, v∈Q;

[0039] Rule 3: Only retain edge weights w(u,v) in the edge weight set E that satisfy u,v∈Q, and w'(u,v)=w(u,v);

[0040] Where V is the node in the outer sparse graph, E is the edge in the sparse graph, W is the weight of the edge in the sparse graph, Q is the node in the new graph, E' is the edge in the new graph, and W' is the weight of the edge in the new graph;

[0041] Initialize the coding tree T; the coding tree includes the root node and leaf node v r ; Among them, each point in Q corresponds to a leaf node;

[0042] For the new graph G'=(Q, E', W'), the coding tree is constructed using the following rules:

[0043] Rule 1: Each node in the coding tree corresponds to a point in Q;

[0044] Rule 2: The root node in the coding tree corresponds to Q;

[0045] Rule 3: For each node v in the encoding tree, the corresponding parent node is v p , the corresponding child node is v c ;

[0046] Rule 4: The intersection of any two child nodes is empty;

[0047] According to the constructed coding tree, the structural entropy is calculated using the following formula:

[0048]

[0049] Where VV is the point set; vol(v) is the weight sum of the leaf nodes in the child node set of the coding tree node v; VOL is the weight sum of all edges;

[0050] According to the obtained structural entropy, the following adjustment rules are used to adjust the nodes of the coding tree so that the structural entropy is as small as possible under the condition of a specific tree height:

[0051] Rule a: Traverse all possible combinations, select the two child nodes with the largest reduction in structural entropy, merge the two child nodes into a new tree node, and use the two child nodes as the child nodes of the new tree node; repeat this step until only two child nodes remain on the root node;

[0052] Rule b: Traverse all non-leaf nodes that do not meet the tree height limit, select the non-leaf node that increases the least in structural entropy after deletion, delete it, and make the child node of the deleted leaf node the child node of the parent node of the deleted leaf node; repeat this step until the tree height meets the set requirements.

[0053] Step S5 calculates the structural entropy of the leaf nodes of the data space coding tree obtained in step S4 to obtain the score of each data point in the feature point set, and obtains the score of each data point in the data set to be detected based on the score and mapping relationship of each data point, which specifically includes the following steps:

[0054] According to the coding tree corresponding to the new graph G'=(Q, E', W') obtained in step S4, the sum of the structural entropies of each data point in the new graph G'=(Q, E', W') is calculated; the sum of the structural entropies is the sum of the structural entropies of all nodes on the path from the leaf node corresponding to the data point to the root node;

[0055] The sum of the structural entropies of each data point in the new graph G'=(Q,E',W') is used as the structural entropy of the corresponding data point in the sparse graph G=(V,E,W); the structural entropy of each data point in the sparse graph G=(V,E,W) is used as the score of the corresponding data point in the data set to be tested.

[0056] Step S6, based on the scores of each data point in the data set to be detected obtained in step S5, completes the detection of abnormal data in the data set to be detected. Specifically, it includes the following steps:

[0057] Identify data points with scores in the top several places (e.g. 10%) as abnormal candidate points;

[0058] Combined with other anomaly detection methods, the candidate set is further screened to complete abnormal data detection.

[0059] The present invention also provides a system for realizing the method for detecting abnormal data in the data set, comprising a data acquisition module, a sparse graph construction module, a point set generation module, a coding tree generation module, a score calculation module and an anomaly detection module; the data acquisition module, the sparse graph construction module, the point set generation module, the coding tree generation module, the score calculation module and the anomaly detection module are connected in series in sequence; the data acquisition module is used to acquire the data set to be detected and upload the data information to the sparse graph construction module; the sparse graph construction module is used to perform sparse processing on the acquired data set according to the received data information to construct a sparse graph, and upload the data information to the point set generation module; the point set generation module is used to sample and map the obtained sparse graph according to the received data information to generate an abnormal data set. The feature point set is obtained and the data information is uploaded to the coding tree generation module; the coding tree generation module is used to calculate the corresponding structural entropy information according to the received data information and the obtained feature point set, and generate a data space coding tree, and upload the data information to the score calculation module; the score calculation module is used to calculate the structural entropy of the leaf nodes of the obtained data space coding tree according to the received data information to obtain the score of each data point in the feature point set, and obtain the score of each data point in the data set to be detected according to the score and mapping relationship of each data point, and upload the data information to the anomaly detection module; the anomaly detection module is used to complete the detection of abnormal data in the data set to be detected according to the received data information and the obtained score of each data point in the data set to be detected.

[0060] The present invention also provides a data cleaning method including the method for detecting abnormal data in the data set, comprising the following steps:

[0061] (1) Obtain the data set to be cleaned;

[0062] (2) using the method for detecting abnormal data in a data set, to detect abnormal data on the data set to be cleaned obtained in step (1);

[0063] (3) According to the abnormal data detection results obtained in step (2), complete the data cleaning of the data set to be cleaned.

[0064] The method, system and data cleaning method for detecting abnormal data in a data set provided by the present invention not only realize the detection and corresponding cleaning of abnormal data in a data set through sparse processing of the data set, sampling mapping, construction of the corresponding coding tree and calculation of structural entropy, but also have higher reliability and better accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 Schematic diagram of the detection method of the present invention.

[0066] Figure 2 Schematic diagram of the functional modules of the system of the present invention.

[0067] Figure 3 Schematic diagram of the data cleaning method of the present invention. DETAILED DESCRIPTION

[0068] like Figure 1 The following is a flow chart of the detection method of the present invention: The method for detecting abnormal data in a data set provided by the present invention comprises the following steps:

[0069] S1. Obtain the data set to be tested;

[0070] S2. Performing sparse processing on the data set obtained in step S1 to construct a sparse graph; specifically comprising the following steps:

[0071] Initialize the core point set to a randomly selected point and iteratively construct the core point set: calculate each data point x i The distance to all core points in the core point set, and the distance to the nearest core point is set as the data point x i The distance to the core point set is to add the data point farthest from the current core point set to the core point set; repeat this step until the distance from all data points to the nearest core point is less than the set value ε, and the construction of the core point set is completed;

[0072] Calculate the median of the distance between all data points and the closest core point, and set the median as the distance threshold ε1;

[0073] Calculate for each data point x i The distance to other data points in the data set, setting the neighbor parameter N, gets the distance x from each data point i The set of the most recent N data points {xi,1 ,x i,2 ,...,x i,N}, set the point in the set to be the data point x i Neighbor points;

[0074] Filter each data point x according to the set distance threshold ε1 i Only the edges with a distance less than the threshold ε1 to its neighboring points are retained;

[0075] Finally, a sparse graph is obtained;

[0076] S3. Sampling and mapping the sparse graph obtained in step S2 to generate a feature point set; specifically comprising the following steps:

[0077] According to the sparse graph obtained in step S2, all points in the sparse graph are initialized to the point set X = {x1, x2, ..., x n}, and as the current point set, n is the total number of points;

[0078] The mapping σ of the initialization point set is for each data point x i Map to itself;

[0079] Repeat steps A to E until the set conditions are met:

[0080] A. From the current point set X i In the process, random sampling with replacement is performed to obtain a point set S of size αt i , where α is the set sampling parameter and t is the set sample cardinality parameter;

[0081] B. Calculate each point in the current point set to point set S i The distance of each point in the point set S and record the distance of each point in the current point set in the point set S i The closest point in the middle;

[0082] C. According to the current point set X i , point set S i And the set proportional parameter β, the minimum radius ρ is calculated using the following formula i :

[0083] |B(S i ,X i ,ρ i )|≥β|X i |

[0084] Where B(S i ,X i ,ρ i ) represents the current point set X i Each point and point set S iThe distance between the nearest points is less than ρ i The set of points is represented as point set C i ;|B(S i ,X i ,ρ i )| represents the set B(S i ,X i ,ρ i ) the number of midpoints; |X i | represents the set X i the number of midpoints;

[0085] D. According to point set C i , the point set C i Each point in is mapped to the point set S i The closest point in the middle;

[0086] E. In point set S i Remove point set C from i The points in the , and the obtained point set is used as the current point set X i ;

[0087] The setting condition is: current point set X i The number of points in is less than the set value, and the current corresponding number of iterations r is saved;

[0088] Finally, the final current point set X r Each point in is mapped to itself;

[0089] For the current point set X r And the point set S in each iteration i For all elements x in , calculate the number of times x is mapped and use it as the weight ω of element x x ;(x,ω x ) Add to the weight set Q;

[0090] Finally, the set of data points with weights, namely the weight set Q, is set as the feature point set;

[0091] S4. Calculate the corresponding structural entropy information based on the feature point set obtained in step S3 and generate a data space coding tree; specifically, the steps include:

[0092] Based on the weight set Q obtained in step S3 and the sparse graph G = (V, E, W) obtained in step S2, a new graph G' = (Q, E', W') is constructed using the following rules:

[0093] Rule 1: Only keep the data points in the data point set V that exist in the weight set Q;

[0094] Rule 2: Only retain the edges (u, v) in the edge set E that satisfy u, v∈Q;

[0095] Rule 3: Only retain edge weights w(u,v) in the edge weight set E that satisfy u,v∈Q, and w'(u,v)=w(u,v);

[0096] Where V is the node in the outer sparse graph, E is the edge in the sparse graph, W is the weight of the edge in the sparse graph, Q is the node in the new graph, E' is the edge in the new graph, and W' is the weight of the edge in the new graph;

[0097] Initialize the coding tree T; the coding tree includes the root node and leaf node v r ; Among them, each point in Q corresponds to a leaf node;

[0098] The coding tree is a hierarchical tree structure that can reflect the hierarchical abstraction of nonlinear patterns between data points. Based on this structure, features with global structural information can be extracted from the tree, which helps to distinguish outliers from normal values. The standard coding tree is a binary tree constructed from leaf nodes to the root node. In the coding tree, each leaf node represents a single data point of a given dataset, while each non-leaf node represents a small partition of the graph.

[0099] For the new graph G'=(Q, E', W'), the coding tree is constructed using the following rules:

[0100] Rule 1: Each node in the coding tree corresponds to a point in Q;

[0101] Rule 2: The root node in the coding tree corresponds to Q;

[0102] Rule 3: For each node v in the encoding tree, the corresponding parent node is v p , the corresponding child node is v c ;

[0103] Rule 4: The intersection of any two child nodes is empty;

[0104] According to the constructed coding tree, the structural entropy is calculated using the following formula:

[0105]

[0106] Where VV is the point set; vol(v) is the weight sum of the leaf nodes in the child node set of the coding tree node v; VOL is the weight sum of all edges;

[0107] According to the obtained structural entropy, the following adjustment rules are adopted, using the iterative methods COMBINE and DROP proposed by Wu et al. to adjust the nodes of the coding tree so that the structural entropy is as small as possible under the condition of a specific tree height:

[0108] Rule a: Traverse all possible combinations, select the two child nodes with the largest reduction in structural entropy, merge the two child nodes into a new tree node, and use the two child nodes as the child nodes of the new tree node; repeat this step until only two child nodes remain on the root node;

[0109] Rule b: Traverse all non-leaf nodes that do not meet the tree height limit, select the non-leaf node that increases the least in structural entropy after deletion, delete it, and make the child node of the deleted leaf node the child node of the parent node of the deleted leaf node; repeat this step until the tree height meets the set requirements;

[0110] S5. Calculate the structural entropy of the leaf nodes of the data space coding tree obtained in step S4 to obtain the score of each data point in the feature point set, and obtain the score of each data point in the data set to be detected based on the score of each data point and the mapping relationship; specifically, the following steps are included:

[0111] According to the coding tree corresponding to the new graph G'=(Q, E', W') obtained in step S4, the sum of the structural entropies of each data point in the new graph G'=(Q, E', W') is calculated; the sum of the structural entropies is the sum of the structural entropies of all nodes on the path from the leaf node corresponding to the data point to the root node;

[0112] The sum of the structural entropies of each data point in the new graph G'=(Q,E',W') is used as the structural entropy of the corresponding data point in the sparse graph G=(V,E,W); the structural entropy of each data point in the sparse graph G=(V,E,W) is used as the score of the corresponding data point in the data set to be tested;

[0113] S6. Based on the scores of each data point in the data set to be detected obtained in step S5, complete the detection of abnormal data in the data set to be detected; specifically, the following steps are included:

[0114] Identify data points with scores in the top several places (e.g. 10%) as abnormal candidate points;

[0115] In addition, in order to obtain more reliable results, other anomaly detection methods can be combined to further screen the candidate set and complete abnormal data detection.

[0116] The effect of the detection method of the present invention is further described below with reference to an embodiment:

[0117] The detection method of the present invention was compared with the existing EOCD algorithm, DIF algorithm, ROD algorithm, LUNAR algorithm and COPOD algorithm on 15 data sets. The detailed information of the data sets is shown in Table 1:

[0118] Table 1 Dataset information diagram

[0119] serial number Dataset Number of points Dimensions Number of outliers 1 Hepatitis 80 19 13 2 wine 129 13 10 3 WPBC 198 33 47 4 glass 214 7 9 5 vertebral 240 6 30 6 WDBC 367 30 10 7 yeast 1484 8 507 8 fault 1941 27 673 9 SpamBase 4207 57 1679 10 Wilt 4819 5 257 11 optdigits 5216 64 150 12 landsat 6435 36 1333 13 zelnik2 303 2 106 14 Cure-T2-4K 4200 2 200 15 Cluto-t7-10k 10000 2 792

[0120] The EOCD algorithm is the algorithm proposed in the paper "Ecod: Unsupervised outlier detection using empirical cumulative distribution functions" published by Li et al. in 2022; the DIF algorithm is the algorithm proposed in the paper "Deep isolation forest for anomaly detection" published by Xu et al. in 2023; the ROD algorithm is the algorithm proposed in the paper "A novel outlier detection method for multivariate data" published by Almardeny et al. in 2020; the LUNAR algorithm is the algorithm proposed in the paper "Lunar: Unifying local outlier detection methods via graph neural networks" published by Goodge et al. in 2022; the COPOD algorithm is the algorithm proposed in the paper "COPOD: Copula-based outlier detection" published by Li et al. in 2020;

[0121] All experimental codes are implemented in Python. The implementation environment is: the operating system is 64-bit Ubuntu 20.04.6LTS server version; the processor model is 72Intel Xeon Gold 6230; the memory size is 500GB;

[0122] In the experiment, following the settings of Li et al. (2022), five external evaluation scores were used: precision (prec), recall (recall), F1 score (f1), ROC score (roc), and average precision (ap). Precision measures the accuracy of correctly identifying true outliers; recall measures the ability of the outlier detector to correctly identify true outliers; F1 score is the harmonic mean of precision and recall, providing a balance between the two metrics; ROC score and average precision score are both used to evaluate the performance of the prediction model, where the ROC score provides an overview of the classification ability, while the average precision score focuses on the model's performance in terms of precision and recall.

[0123] Finally, the experimental data are shown in Table 2, Table 3, and Table 4: In the tables, higher scores indicate better anomaly detection performance;

[0124] Table 2 The first schematic diagram of experimental comparison data

[0125]

[0126] Table 3 Second schematic diagram of experimental comparison data

[0127]

[0128] Table 4 The third schematic diagram of experimental comparison data

[0129] Dataset index The present invention ECOD DIF ROD LUNAR COPOD Hepatitis f1 0.2857 0.1904 0.0952 0.1904 0 0.1904 wine f1 0.909 0.2608 0 0.4347 0 0.5217 WPBC f1 0.2121 0.0597 0.0895 0.0895 0.0895 0.1194 glass f1 0.4 0.1333 0.1935 0.1333 0.1935 0.1935 vertebral f1 0.3703 0.037 0.074 0.074 0 0 WDBC f1 0.4347 0.3829 0.2127 0.4347 0.3829 0.4347 yeast f1 0.2351 0.1615 0.1158 0.1737 0.1006 0.1493 fault f1 0.2998 0.152 0.1868 0.0738 0.2791 0.1176 SpamBase f1 0.3992 0.22 0.1419 0.2609 0.1542 0.2276 Wilt f1 0.1869 0.0378 0.0054 0.0243 0.0108 0.0108 optdigits f1 0.1579 0.0327 0.0208 0.0892 0.0238 0.0357 landsat f1 0.3593 0.1031 0.005 0 0.1861 0.1021 zelnik2 f1 0.4525 0.4525 0.4525 0.4525 0.4525 0.4525 Cure-T2-4K f1 0.4709 0.1 0.0935 0.0677 0.4032 0.0612 Cluto-t7-10k f1 0.4843 0.2689 0.1104 0.1529 0.4754 0.27

[0130] Tables 2 to 4 show that our proposed solution achieves an average 28.91% improvement in accuracy over state-of-the-art results. For specific datasets, our proposed solution achieves 80.78% and 55.22% improvements over the ECOD and ROD algorithms, respectively. Recall improves by 27.54% overall; for specific datasets, our proposed solution achieves 78.79% and 53.53% improvements over ECOD and ROD, respectively. Our proposed solution improves the F1 score by an average of 28.24%. For specific datasets, our proposed solution achieves 80.78%, 56.51%, 40.46%, and 81.76% improvements over ECOD, ROD, LUNAR, and COPOD, respectively. Our proposed solution achieves an average 9.50% improvement in AP over state-of-the-art results. For specific datasets, our proposed solution achieves 63.88%, 42.93%, 74.20%, and 84.99% improvements over ECOD, ROD, LUNAR, and COPOD, respectively. In terms of ROC scores, our solution is slightly inferior to the state-of-the-art results. However, compared to the ECOD, DIF, LUNAR, and COPOD algorithms, our solution achieves overall F1 scores that are 18.56%, 29.23%, 14.81%, and 18.32%, respectively. These results demonstrate that our solution significantly improves anomaly detection quality compared to traditional anomaly detection algorithms such as ECOD, ROD, LUNAR, and COPOD.

[0131] like Figure 2The figure shows a schematic diagram of the functional modules of the system of the present invention: the system disclosed in the present invention for realizing the method for detecting abnormal data in the data set includes a data acquisition module, a sparse graph construction module, a point set generation module, a coding tree generation module, a score calculation module and an anomaly detection module; the data acquisition module, the sparse graph construction module, the point set generation module, the coding tree generation module, the score calculation module and the anomaly detection module are connected in series in sequence; the data acquisition module is used to acquire the data set to be detected and upload the data information to the sparse graph construction module; the sparse graph construction module is used to perform sparse processing on the acquired data set according to the received data information to construct a sparse graph, and upload the data information to the point set generation module; the point set generation module is used to sample the obtained sparse graph according to the received data information and mapping to generate a feature point set, and upload the data information to the coding tree generation module; the coding tree generation module is used to calculate the corresponding structural entropy information according to the received data information and the obtained feature point set, and generate a data space coding tree, and upload the data information to the score calculation module; the score calculation module is used to calculate the structural entropy of the leaf nodes of the obtained data space coding tree according to the received data information to obtain the score of each data point in the feature point set, and obtain the score of each data point in the data set to be detected according to the score and mapping relationship of each data point, and upload the data information to the anomaly detection module; the anomaly detection module is used to complete the detection of abnormal data in the data set to be detected according to the received data information and the obtained score of each data point in the data set to be detected.

[0132] like Figure 3 The data cleaning method disclosed in the present invention includes the following steps:

[0133] (1) Obtain the data set to be cleaned;

[0134] (2) using the method for detecting abnormal data in a data set, to detect abnormal data on the data set to be cleaned obtained in step (1);

[0135] (3) According to the abnormal data detection results obtained in step (2), complete the data cleaning of the data set to be cleaned.

[0136] In addition, the abnormal data detection method and the corresponding data cleaning method of the present invention have broad application potential in multiple occasions in different fields, such as fraud analysis in the financial field, health monitoring in the healthcare field, intrusion detection in the information security field, and equipment anomaly detection in the industrial production field.

Claims

1. A method for detecting abnormal data in a data set, comprising the following steps: S1. Obtain the data set to be tested; S2. performing sparse processing on the data set obtained in step S1 to construct a sparse graph; S3. Sampling and mapping the sparse graph obtained in step S2 to generate a set of feature points; S4. Calculate the corresponding structural entropy information based on the feature point set obtained in step S3 and generate a data space coding tree; S5. Calculate the structural entropy of the leaf nodes of the data space coding tree obtained in step S4 to obtain the score of each data point in the feature point set, and obtain the score of each data point in the data set to be detected based on the score and mapping relationship of each data point; S6. Complete the detection of abnormal data in the data set to be detected based on the score of each data point in the data set to be detected obtained in step S5.

2. The method for detecting abnormal data in a data set according to claim 1, characterized in that Step S2, in which the data set obtained in step S1 is subjected to sparse processing to construct a sparse graph, specifically includes the following steps: Initialize the core point set to a randomly selected point and iteratively construct the core point set: calculate each data point x i The distance to all core points in the core point set, and the distance to the nearest core point is set as the data point x i The distance to the core point set, add the data point farthest from the current core point set to the core point set; Repeat this step until the distance from all data points to the nearest core point is less than the set value ε, completing the construction of the core point set; Calculate the median of the distance between all data points and the closest core point, and set the median as the distance threshold ε1; Calculate for each data point x i The distance to other data points in the data set, setting the neighbor parameter N, gets the distance x from each data point i The set of the most recent N data points {x i,1 ,x i,2 ,...,x i,N }, set the point in the set to be the data point x i Neighbor points; Filter each data point x according to the set distance threshold ε1 i Only the edges with a distance less than the threshold ε1 to its neighboring points are retained; Finally, a sparse graph is obtained.

3. The method for detecting abnormal data in a data set according to claim 2, characterized in that Step S3 of sampling and mapping the sparse graph obtained in step S2 to generate a feature point set specifically includes the following steps: According to the sparse graph obtained in step S2, all points in the sparse graph are initialized to the point set X = {x1, x2, ..., x n }, and as the current point set, n is the total number of points; The mapping σ of the initialization point set is for each data point x i Map to itself; Repeat steps A to E until the set conditions are met: A. From the current point set X i In the process, random sampling with replacement is performed to obtain a point set S of size αt i , where α is the set sampling parameter and t is the set sample cardinality parameter; B. Calculate each point in the current point set to point set S i The distance of each point in the point set S and record the distance of each point in the current point set in the point set S i The closest point in the middle; C. According to the current point set X i , point set S i And the set proportional parameter β, the minimum radius ρ is calculated using the following formula i : |B(S i ,X i ,r i )|≥β|X i | Where B(S i ,X i ,ρ i ) represents the current point set X i Each point and point set S i The distance between the nearest points is less than ρ i The set of points is represented as point set C i ;|B(S i ,X i ,ρ i )| represents the set B(S i ,X i ,ρ i ) the number of midpoints; |X i | represents the set X i the number of midpoints; D. According to point set C i , the point set C i Each point in is mapped to the point set S i The closest point in the middle; E. In point set S i Remove point set C from i The points in the , and the obtained point set is used as the current point set X i ; The setting condition is: current point set X i The number of points in is less than the set value, and the current corresponding number of iterations r is saved; Finally, the final current point set X r Each point in is mapped to itself; For the current point set X r And the point set S in each iteration i For all elements x in , calculate the number of times x is mapped and use it as the weight ω of element x x ;(x,ω x ) Add to the weight set Q; Finally, the weight set Q is set as the feature point set.

4. The method for detecting abnormal data in a data set according to claim 3, characterized in that Step S4 calculates the corresponding structural entropy information based on the feature point set obtained in step S3 and generates a data space coding tree, which specifically includes the following steps: Based on the weight set Q obtained in step S3 and the sparse graph G = (V, E, W) obtained in step S2, a new graph G' = (Q, E', W') is constructed using the following rules: Rule 1: Only keep the data points in the data point set V that exist in the weight set Q; Rule 2: Only retain the edges (u, v) in the edge set E that satisfy u, v∈Q; Rule 3: Only retain edge weights w(u,v) in the edge weight set E that satisfy u,v∈Q, and w'(u,v)=w(u,v); Where V is the node in the outer sparse graph, E is the edge in the sparse graph, W is the weight of the edge in the sparse graph, Q is the node in the new graph, E' is the edge in the new graph, and W' is the weight of the edge in the new graph; Initialize the coding tree T; the coding tree includes the root node and leaf node v r ; Among them, each point in Q corresponds to a leaf node; For the new graph G'=(Q, E', W'), the coding tree is constructed using the following rules: Rule 1: Each node in the coding tree corresponds to a point in Q; Rule 2: The root node in the coding tree corresponds to Q; Rule 3: For each node v in the encoding tree, the corresponding parent node is v p , the corresponding child node is v c ; Rule 4: The intersection of any two child nodes is empty; According to the constructed coding tree, the structural entropy is calculated using the following formula: Where VV is the point set; vol(v) is the weight sum of the leaf nodes in the child node set of the coding tree node v; VOL is the weight sum of all edges; According to the obtained structural entropy, the following adjustment rules are used to adjust the nodes of the coding tree so that the structural entropy is as small as possible under the condition of a specific tree height: Rule a: Traverse all possible combinations, select the two child nodes with the largest reduction in structural entropy, merge the two child nodes into a new tree node, and use the two child nodes as the child nodes of the new tree node; repeat this step until only two child nodes remain on the root node; Rule b: Traverse all non-leaf nodes that do not meet the tree height limit, select the non-leaf node that increases the least in structural entropy after deletion, delete it, and make the child node of the deleted leaf node the child node of the parent node of the deleted leaf node; repeat this step until the tree height meets the set requirements.

5. The method for detecting abnormal data in a data set according to claim 4, characterized in that Step S5 calculates the structural entropy of the leaf nodes of the data space coding tree obtained in step S4 to obtain the score of each data point in the feature point set, and obtains the score of each data point in the data set to be detected based on the score and mapping relationship of each data point, which specifically includes the following steps: According to the coding tree corresponding to the new graph G'=(Q, E', W') obtained in step S4, the sum of the structural entropies of each data point in the new graph G'=(Q, E', W') is calculated; the sum of the structural entropies is the sum of the structural entropies of all nodes on the path from the leaf node corresponding to the data point to the root node; The sum of the structural entropies of each data point in the new graph G'=(Q,E',W') is used as the structural entropy of the corresponding data point in the sparse graph G=(V,E,W); the structural entropy of each data point in the sparse graph G=(V,E,W) is used as the score of the corresponding data point in the data set to be tested.

6. The method for detecting abnormal data in a data set according to claim 5, characterized in that Step S6, based on the scores of each data point in the data set to be detected obtained in step S5, completes the detection of abnormal data in the data set to be detected, and specifically includes the following steps: The data points with the highest scores are identified as abnormal candidate points.

7. A system for implementing the method for detecting abnormal data in a data set according to any one of claims 1 to 6, characterized in that It includes a data acquisition module, a sparse graph construction module, a point set generation module, a coding tree generation module, a score calculation module and an anomaly detection module; the data acquisition module, the sparse graph construction module, the point set generation module, the coding tree generation module, the score calculation module and the anomaly detection module are connected in series in sequence; the data acquisition module is used to obtain the data set to be detected and upload the data information to the sparse graph construction module; The sparse graph construction module is used to perform sparse processing on the acquired data set according to the received data information to construct a sparse graph, and upload the data information to the point set generation module; The point set generation module is used to sample and map the obtained sparse graph according to the received data information to generate a feature point set, and upload the data information to the coding tree generation module; The coding tree generation module is used to calculate the corresponding structural entropy information based on the received data information and the obtained feature point set, and generate a data space coding tree, and upload the data information to the score calculation module; The score calculation module is used to calculate the structural entropy of the leaf nodes of the data space coding tree based on the received data information to obtain the score of each data point in the feature point set, and obtain the score of each data point in the data set to be detected based on the score and mapping relationship of each data point, and upload the data information to the anomaly detection module; The anomaly detection module is used to complete the detection of anomaly data in the data set to be detected based on the received data information and the obtained scores of each data point in the data set to be detected.

8. A data cleaning method comprising the method for detecting abnormal data in a data set according to any one of claims 1 to 6, characterized in that The steps include: (1) Obtain the data set to be cleaned; (2) using the method for detecting abnormal data in a data set according to any one of claims 1 to 6 to perform abnormal data detection on the data set to be cleaned obtained in step (1); (3) According to the abnormal data detection results obtained in step (2), complete the data cleaning of the data set to be cleaned.

Images