A method and system for identifying and repairing abnormalities in unbalanced data

Abstract

The present invention belongs to the field of data governance technology, and specifically relates to a method and system for identifying and repairing anomalies of unbalanced data. The method includes three parts: cluster analysis of unbalanced mixed data, identification and feature location of abnormal data, and repair of abnormal data. First, a hybrid clustering algorithm based on cost optimization is proposed to solve the problem of effective clustering of mixed data under unbalanced data sets, and to achieve high similarity of data within the same clustering cluster; secondly, an abnormal feature location algorithm based on isolation forest is proposed to solve the problem that the previous unsupervised anomaly detection algorithm can only locate abnormal records but cannot identify specific abnormal features in the records, and to achieve accurate location of problem data; finally, a data repair model based on heterogeneous variational autoencoders is proposed to achieve credible repair of mixed data anomalies, and to solve the problem that the previous artificial intelligence methods are too disturbed by data distribution and noise data during the data repair process, resulting in low credibility of repaired data.

Description

Technical Field

[0001] The present invention belongs to the technical field of data governance, and specifically relates to a method and system for identifying and repairing anomalies in unbalanced data. Background Art

[0002] In the field of data governance, intelligent and accurate identification and location of abnormal data in the data to be governed has always been a key requirement. The ability to intelligently repair the identified abnormal data has put forward higher requirements for data governance work in the context of improving the quality of data supply and building a data factor market.

[0003] In existing data governance solutions, the most common way to identify and repair abnormal data is for data owners to compile relevant data governance rules based on business understanding to identify abnormal data. For repairing abnormal data, the problematic data is usually returned to the data production department for manual correction by business personnel. In recent years, with the development of artificial intelligence technology, many anomaly detection algorithms have been applied to the field of data governance for automatic identification of abnormal data, and occasionally there are studies on the use of neural networks to repair abnormal data. However, various existing anomaly detection methods are based on the premise that the data to be governed obeys a certain distribution. Therefore, in reality, the anomaly detection and data repair capabilities of related algorithms are easily affected by data distribution and noise data, and the effect is far less than expected.

[0004] The goal of this invention is to make the existing manual work of abnormal data screening and repair intelligent and automated through the innovation of algorithms and models, and to solve the shortcomings of the existing solutions in abnormality identification, problem repair, etc. Through analysis, it can be seen that the key problems of the existing solutions are as follows:

[0005] Unable to effectively deal with the problem of data imbalance. Classic abnormal data identification algorithms usually act directly on the overall data, and their premise is that the data obeys a certain distribution. However, in reality, the diversity of business leads to differences in the generated data, resulting in data imbalance, especially data is often mixed data with both categorical and numerical types. When processing unbalanced data, due to the imbalance of data categories, data of different categories often obey different distributions, resulting in the classic abnormal data identification algorithm can only passively adapt to the overall distribution of data in the global modeling process, and cannot accurately capture anomalies with finer granularity distribution, which seriously affects the effect and may lead to misjudgment and reduce the accuracy of anomaly identification. In view of this, cluster analysis before imbalanced data anomaly identification is an effective solution. However, some existing cluster analysis methods, such as K-Means, DBSCAN, K-Mode, K-Prototype, etc., cannot effectively process mixed data. These methods cannot effectively take into account both numerical and categorical data when dividing groups, resulting in limited usage scenarios.

[0006] The problem of abnormal feature location cannot be achieved. Data governance has to deal with massive and unpredictable data. Therefore, anomaly identification and location cannot be achieved through supervised learning algorithms, because this requires a large amount of labeled data for training and modeling, which is impossible in reality. Some existing unsupervised anomaly detection algorithms, such as local outlier factor (LOF) and isolation forest (IF) algorithms, can usually only give record-level anomaly assessments even if the data meets the premise assumptions. That is, they can only clearly identify which data is problematic, but cannot locate which specific feature (field) value in the abnormal record caused the anomaly. This makes the algorithm less interpretable and cannot perform targeted repairs on abnormal features when repairing abnormal data.

[0007] The problem of being unable to provide efficient and reliable abnormal data repair capabilities. In the existing data governance process, most anomaly detection methods do not have the ability to repair anomalies. They still use traditional rule-driven manual repair based on human experience values, which cannot meet the repair needs of massive mixed business data. There is an urgent need for an automated anomaly repair method. Although deep learning and other methods are occasionally used to try to repair anomalies, the relevant methods are based on the premise that the training samples meet the same distribution, and the repair effect is easily disturbed by abnormal data and noise data in the data, resulting in repair distortion. For example, for a data set, for most of its records, a certain numerical feature has a value between 20 and 100, but occasionally there are very few extreme value data exceeding 100,000, resulting in the mean of the overall feature exceeding 1000. At this time, the correct repair result should be to repair these extreme values ​​to between 20 and 100, but in fact, the existing VAE method often repairs these extreme values ​​to near the mean, which cannot meet the actual needs. Especially in the case of unbalanced mixed data, the data repair effect of the relevant model is even worse. Summary of the invention

[0008] In order to overcome the problems in the prior art, the present invention proposes a method and system for identifying and repairing anomalies in unbalanced data.

[0009] The technical solution of the present invention to solve the above technical problems is as follows:

[0010] In a first aspect, the present invention provides a method for identifying and repairing anomalies in unbalanced data, comprising the following steps:

[0011] Step 1. Identify the feature types of the input unbalanced mixed data, where the feature types include: categorical type and numerical type;

[0012] Step 2. Construct a model based on the cost-optimized hybrid clustering algorithm, comprehensively consider the differences between the features of the unbalanced mixed data, perform cluster analysis on the unbalanced mixed data, and thus divide the unbalanced mixed data into different clusters;

[0013] Step 3. Build a model based on the abnormal feature location algorithm of the isolation forest to identify the abnormal data in each cluster after clustering and locate its specific abnormal features;

[0014] Step 4. Build a data repair model based on heterogeneous variational autoencoder to accurately repair mixed data anomalies and missing data.

[0015] Furthermore, the step 2 specifically includes:

[0016] Step 2-1: Data initialization; feature processing and feature information statistics, calculation of feature weights;

[0017] Step 2-2: Cluster splitting: Split the unbalanced mixed data based on the splitting strategy, split the data into two new clusters, and count the numerical and categorical feature information within the cluster;

[0018] Step 2-3: Iteratively allocate data points; redistribute the overall data and regenerate clusters;

[0019] Step 2-4: Calculate the intra-cluster cost; for the regenerated cluster, calculate its intra-cluster cost, which is defined as the distance from all points in the cluster to the cluster distribution; calculate the total cost of each cluster, and find the cluster with the largest cost as the cluster to be split;

[0020] Step 2-5: Iterative splitting: repeat the above steps until the maximum number of splits is reached, and each split generates a new cluster;

[0021] Step 2-6: Cluster merging; merging similar clusters.

[0022] Furthermore, the step 3 comprises:

[0023] Step 3-1: Feature group division: obtain cluster labels after clustering and continuous feature data, and divide the feature data into different groups according to the cluster labels;

[0024] Step 3-2: Construct an abnormal feature location model based on isolation forest for each feature group;

[0025] Step 3-3: Calculate the anomaly score; for each feature data, calculate its path depth in each isolated tree, that is, the number of edges that the feature data passes through from the root node of the tree to the leaf node; then calculate the average path length of the feature data in all trees, normalize the average path length of all data points, and obtain the anomaly score of each data; at the same time, in the process of calculating the anomaly score of each data, retain the calculation result of the data on each tree and keep it as a weight score for subsequent use;

[0026] Step 3-4: Locate abnormal features;

[0027] Step 3-5: Result output: Based on steps 3-2 to 3-4, the abnormal feature positioning results of each feature group are obtained, and the abnormal label and positioning results are output according to the set threshold.

[0028] Furthermore, in step 3-2, the abnormal feature localization model based on isolation forest is constructed, which specifically includes:

[0029] Step 3-2-1: Setting the positioning model parameters, including the number of isolated trees to be constructed, the number of samples for each isolated tree, the degree of freedom of the segmentation hyperplane, and the maximum depth of the isolated tree;

[0030] Step 3-2-2: According to the set sampling number, for each isolated tree, randomly select a specified number of samples from the data to be managed by replacing the sampling method, set a root node, and use the root node as the current node;

[0031] Step 3-2-3: According to the set splitting hyperplane degrees of freedom, randomly select several features of the set hyperplane degrees of freedom on the current node as target features for splitting, and construct the left subtree and right subtree of the node respectively according to the splitting results; at the same time, save the target feature information of the current node;

[0032] Step 3-2-4: Construct an isolation tree;

[0033] Iterate step 3-2-3 in the child node until the child node cannot be further split or the child node has reached the maximum depth of the tree set in step 3-2-1, then stop iterating and complete the construction of an isolated tree;

[0034] Step 3-2-5: Extract segmentation path information;

[0035] Based on step 3-2-3, extract and save the target feature information selected on the entire path from the root node to the leaf node;

[0036] Step 3-2-6: Construct an isolation forest;

[0037] Based on the number of isolated trees required to be constructed set in step 3-2-1, steps 3-2-2 to 3-2-5 are executed in a loop until the construction of all isolated trees and the extraction of path target feature information are completed.

[0038] Furthermore, locating abnormal features in step 3-4 specifically includes:

[0039] Step 3-4-1: Based on step 3-2, obtain the segmentation path information of each data in each tree and the depth of the tree finally reached, and extract the target feature information of the data in the middle node of each layer of the path according to the path information;

[0040] Step 3-4-2: Obtain target feature information based on step 3-4-1, and calculate the limit depth based on the maximum depth of the tree set in step 3-2-1; and divide the target feature information of each data on all trees into two parts according to the calculated limit depth, and calculate the frequency information of the target features involved in each part of the tree where the depth is less than the limit depth;

[0041] Step 3-4-3: Multiply the weight score calculated in step 3-3 by the frequency information calculated in step 3-4-2, and add them up according to the limit depth in 3-4-2 and take the average. The weight score of each piece of data needs to be divided into two parts according to the limit depth.

[0042] Step 3-4-4: Calculate the comprehensive score of characteristic abnormalities.

[0043] Further, step 4 includes: step 4-1: data feature transformation; based on step 1, the original mixed data and data structure information are obtained, and the feature transformation is performed on the original data to convert the original data into feature data acceptable to the neural network;

[0044] Based on the data structure information identified in step 1, the classification and numerical features in the original data are obtained. For the classified data, the classification data is encoded by encoding technology, the classification data is converted into numerical representation, and the encoding information of the classification data is saved;

[0045] Step 4-2: Set the mask structure; according to the abnormal identification and abnormal feature location results of step 3, mask the feature data identified as abnormal or missing; at the same time, adopt a random masking strategy, set the mask threshold percentage, and randomly mask each feature at a certain ratio. Combining the two masking processes, determine the mask structure that is finally input into the network;

[0046] Step 4-3: Build a data repair model based on heterogeneous variational autoencoder; based on the mask structure of step 4-2, mask the feature data, reconstruct the masked data by learning the distribution of non-masked data, so as to achieve the effect of repairing outliers;

[0047] Step 4-4: abnormal data repair; based on step 4-3, the network is constructed and trained. Through the training optimization of the recognition model and the generation model in the network, the distribution of each dimension data generated by the network is continuously close to the real distribution until convergence; then, the mask data is reconstructed based on the trained model, and the reconstructed value is used as the abnormal repair recommendation value, so as to achieve the purpose of abnormal data repair;

[0048] Step 4-5: Data feature restoration, restore the repair result output from step 4-4 to the format of the original data; first, in step 4-3, the input data is regularized, and the output result needs to be inversely encoded into the format of the network input data; secondly, in step 4-1, different types of data are preprocessed accordingly, and then inversely processed to convert them into the format of the original data.

[0049] In a second aspect, the present invention also provides an abnormality identification and repair system for imbalanced data, including: a data management module, a feature processing module, a model training module, a model management module, a feature restoration module, and a result management module;

[0050] The data management module is used to perform metadata management and data integration management on the data to be governed. It is responsible for organizing the required governance data and initially identifying the data structure based on the metadata management function.

[0051] The feature processing module is used to transform the data to be managed and convert it into feature data that can be directly input into various models;

[0052] The model training module is used to manage the model training and prediction process, including three sub-modules: data clustering, abnormal feature location, and abnormal repair;

[0053] The data clustering submodule is used to obtain the processed feature data from the feature processing module, perform clustering analysis on the unbalanced mixed data based on the cost-optimized hybrid clustering algorithm, divide the data to be managed into different cluster groups, and finally output the feature values ​​and cluster group division labels that make the classified data continuous during the clustering process;

[0054] The abnormal feature location submodule obtains continuous feature data and group division labels from the mixed data clustering submodule; based on the group division labels, for each group of data, a certain number or proportion of data is randomly selected using the abnormal feature location algorithm based on the isolation forest to train an abnormal location model for each group, and the feature data of each group is tested based on the trained model to output the final abnormal feature location result;

[0055] The anomaly repair submodule obtains the anomaly recognition and anomaly location results of each group from the anomaly feature location submodule, and trains the data repair model based on the heterogeneous variational autoencoder based on the original data input obtained from the data management module, reconstructs and repairs the abnormal data based on the trained model, and outputs the repair results;

[0056] The model management module is used to manage the models generated by the model training and prediction modules, and to cache and load the models, so that the models can be loaded directly from the cache when processing the same table;

[0057] The feature restoration module is used to reverse the results output by the model training and prediction modules to convert them into the data structure of the original data, specifically, to reverse the processing process of the original data according to the feature processing module;

[0058] The result management module manages the results output by the feature restoration module, including the management of mixed clustering, abnormal feature location, and abnormal repair. It outputs the cluster number and abnormal evaluation value of each data. For data evaluated as abnormal, it outputs the abnormal features corresponding to the abnormal data record and the value after model repair.

[0059] Compared with the prior art, the present invention has the following technical effects:

[0060] In view of the fact that the existing abnormal data identification and repair methods in the field of data governance have high requirements on prerequisites such as data distribution, low anomaly identification accuracy on unbalanced data, and poor anomaly repair credibility, the present invention proposes an anomaly identification and repair method for unbalanced data. The method mainly includes three parts: clustering analysis of unbalanced mixed data, abnormal data identification and feature location, and abnormal data repair. Corresponding to these three parts, a cost-optimized mixed data clustering algorithm (CoMDC) is firstly proposed to solve the effective clustering problem of mixed data under unbalanced data sets, so as to achieve high similarity of data in the same cluster; secondly, an isolation forest based exception feature location algorithm (IFEL) is proposed to solve the problem that the previous unsupervised anomaly detection algorithm can only locate abnormal records but cannot identify the specific abnormal feature fields (or which fields) in the records, so as to achieve accurate location of problem data; finally, a data repair model based on heterogeneous variational autoencoder is proposed. The proposed artificial intelligence (AI) autoencoder (HVAE) can realize the reliable repair of mixed data anomalies and solve the problem that the previous artificial intelligence methods are too disturbed by data distribution and noise data during the data repair process, resulting in low credibility of the repaired data.

[0061] First, a hybrid data clustering algorithm based on cost optimization is proposed to solve the problem that the imbalance of the data to be managed leads to poor abnormal data identification and repair effects. Based on this algorithm, the data in the unbalanced data set can be clustered through operations such as split-aggregation, so that the data is divided into different groups, so that the data in each cluster has a similar pattern, thereby providing a good data basis for subsequent abnormal identification and data repair. In addition, for the most common problem of hybrid data clustering difficulties in data governance, the present invention proposes a new clustering measurement strategy, which calculates the clustering cost based on probability distance, comprehensively considers the degree of difference between classification features and numerical features, and finally aggregates highly similar data into the same cluster. Compared with traditional clustering algorithms that can only support numerical data (such as K-Means), clustering algorithms that can only support classified data (such as K-Mode), and the currently widely used K-Prototype algorithm that supports hybrid attribute clustering, the clustering algorithm proposed in the present invention has a wider range of application and the clustering results are more reasonable.

[0062] Secondly, an abnormal feature location algorithm based on isolation forest is proposed to identify and locate abnormal data in each cluster. The classic isolation forest algorithm has many shortcomings in the application of data governance. For example, it can only be applied to the anomaly detection of numerical data and the anomaly recognition effect is not good under high-dimensional data sets. In addition, the classic isolation forest algorithm can only identify abnormal records, but cannot specifically determine which feature (field) of the abnormal record causes the anomaly. In view of the above-mentioned shortcomings of the classic isolation forest, the abnormal feature location algorithm based on isolation forest proposed in the present invention is an improvement on the classic isolation forest algorithm. On the one hand, it can realize the support of classified data based on a continuous method of classified data in a hybrid data clustering algorithm based on cost optimization. On the other hand, it also provides spatial segmentation based on multiple degrees of freedom, which solves the stability problem of anomaly detection of the classic isolation forest method in high-dimensional data; further, the accurate location of abnormal features (fields) is realized through multi-degree-of-freedom spatial segmentation path analysis, which solves the problem that the classic isolation forest algorithm can only give record-level abnormal evaluation but cannot locate specific abnormal features (fields).

[0063] Third, a data repair model based on heterogeneous variational autoencoders is proposed. Within each cluster, the abnormal data identified by the abnormal feature location algorithm based on the isolation forest is repaired. The present invention improves the classic variational autoencoder and proposes a heterogeneous variational autoencoder. The model designs different likelihood models for different data types, processes abnormal and missing data through mask structures and decomposition decoders, and designs a hierarchical structure to capture the correlation between features, thereby achieving reliable repair of mixed abnormalities and missing data. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 A schematic diagram of a hybrid data clustering algorithm based on cost optimization proposed by the present invention;

[0066] Figure 2 This is a schematic diagram of the structure of the abnormal feature localization algorithm based on isolation forest proposed by the present invention;

[0067] Figure 3 This is a schematic diagram of the overall structure of the data repair model based on the heterogeneous variational autoencoder proposed in the present invention;

[0068] Figure 4The overall flow chart of the method proposed by the present invention;

[0069] Figure 5 The following is a flowchart of the system workflow of the present invention. DETAILED DESCRIPTION

[0070] In order to further explain the technical means and effects taken by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the technical solutions proposed by the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments. The specific features, structures or characteristics in one or more embodiments may be combined in any suitable form. Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by technicians in the technical field of the present invention.

[0071] In one embodiment of the present invention, a method for identifying and repairing anomalies in unbalanced data is provided, comprising the following steps:

[0072] Step 1: Identify the feature types of the input unbalanced mixed data, where the feature types include: categorical type and numerical type.

[0073] In this embodiment, the input data may be structured data of any dimension, and may be pure numerical data, categorical data, or mixed data of both numerical and categorical types.

[0074] It is necessary to identify the types of each feature dimension of the input data. Feature types are divided into two categories: categorical and numerical. The categorical type can be further divided into classification, ordered, etc., and the numerical type can be further divided into integer, real number, positive real number, etc. The feature type identification step is to clearly identify the type of features of each dimension of the input data one by one and divide them into specific feature subdivision categories, so that the subsequent algorithm can correctly interpret and process the data.

[0075] Step 2. Build a model based on the cost-optimized hybrid clustering algorithm, comprehensively consider the differences between the features of the imbalanced mixed data, perform clustering analysis on the imbalanced mixed data, and thus divide the imbalanced mixed data into different clusters.

[0076] This embodiment proposes a mixed data clustering algorithm based on cost optimization, which aims to perform distance-based similarity measurement on unbalanced mixed structured data; the clustering algorithm calculates clustering cost based on probability distance, comprehensively considers the degree of difference between classification and numerical features, and its goal is to divide the entire data set into different clusters according to the different density distributions of different data features, so that highly similar data are aggregated into the same cluster. In this embodiment, not only can numerical data and categorical data be processed at the same time, but also both types of data are standardized to the value range of 0 to 1 based on probability distribution to achieve dedimensionalization, and scientifically assisted by feature weights, optimized clustering is achieved through splitting and fusion iterations based on cost optimization, so that the unbalanced mixed data is divided into different clusters after clustering analysis, and the data in each cluster has a high degree of similarity, eliminating the impact of data imbalance on subsequent algorithms and models.

[0077] Reference Figure 1 , the specific steps are as follows:

[0078] Step 2-1: Initialize the input data;

[0079] Step 2-1-1: Feature processing and feature information statistics;

[0080] Feature processing: For each categorical feature, the probability is used as the new value of the categorical feature to achieve the continuity of the categorical feature. The probability is calculated as follows:

[0081]

[0082] Among them, p ci (s) represents the probability of the eigenvalue s appearing in the ci column, n s It represents the number of occurrences of eigenvalue s, and N is the number of statistical data.

[0083] Feature information statistics: For each numerical feature, calculate the mean and standard deviation of the column and store them for subsequent use; for each categorical feature, calculate its probability value based on the above feature processing method and store it for subsequent use.

[0084] Step 2-1-2: Calculate feature weights;

[0085] Since different features contribute differently to clustering, they need to be distinguished through weight calculation.

[0086] For categorical features, higher information entropy means that the data is more chaotic or uncertain and needs to be distinguished first. The greater the entropy, the greater the weight should be. The weight calculation process is as follows:

[0087]

[0088]

[0089] Among them, E ci represents the information entropy of the categorical features of the i-th column, represents the probability of the eigenvalue s of this column appearing in the data set, |Ci| represents the number of different eigenvalues ​​in the column, which is used to balance the problem that columns with more eigenvalues ​​usually have greater entropy; R ci Represents the feature weight of the categorical feature of the i-th column.

[0090] After calculating the weight of each categorical feature, normalize the overall weight so that the sum of the weights is 1. The calculation formula is:

[0091]

[0092] Among them, W ci represents the normalized weight of the categorical feature of the i-th column, d c Indicates the number of categorical feature columns contained in the data.

[0093] For numerical features, the weight is determined by the variance and the mean. The larger the variance, the smaller the weight. However, since the variance of features with a larger mean is usually larger, the mean is added to balance it. The calculation formula is as follows:

[0094]

[0095] Among them, R ni Represents the weight of the numerical feature in the i-th column, Mean G (ni) and Std G (ni) represent the mean and standard deviation of the column in the data set.

[0096] Similarly, using the above weight normalization method, calculate the numerical normalized weight W ni .

[0097] Step 2-2: cluster splitting;

[0098] The data is split based on the splitting strategy. The splitting process starts with the entire data set, splitting the data into two new clusters, and then each split selects the cluster with the largest cost within the cluster for splitting. The specific process is as follows:

[0099] Step 2-2-1: Select the split point;

[0100] Randomly select a data x from the cluster to be split a As a split point, calculate x based on the distribution probability aThe distance to other data in the cluster. The distance calculation should take into account both categorical features and numerical features, so that the dimensions of categorical feature distance and numerical feature distance are unified, and avoid distance tilt, which may cause a certain feature to have too great an impact on the total distance.

[0101] Calculate the distance of the classification feature, the calculation formula is as follows:

[0102]

[0103]

[0104] Among them, Dis c (x a ,x b ) represents x a and x b The distance between the classification features, is the data x calculated in step 2-1-1 a The probability of the eigenvalue corresponding to the i-th dimension classification column appearing in the overall data; is the data x b The probability of the eigenvalue corresponding to the categorical column of the i-th dimension appearing in the overall data; f(a,b) represents the judgment function, which is used to judge whether a and b are equal.

[0105] For numerical features, the probability density distribution function is used to convert the distance between two points into the probability of the points appearing in the distribution. The larger the probability, the more similar the two points are, that is, the closer the distance is. Since the probability value is between 0 and 1, the distance calculation can be consistent with the categorical feature. The calculation formula is as follows:

[0106]

[0107] Among them, Dis n (x a ,x b ) represents x a and x b The overall distance between numerical features, and Respectively represent x a and x b The value of the i-th dimension column, d n is the number of numerical features, Indicates is the mean, Std G (ni) is the probability density function of the distribution generated by the standard deviation, express The probability of occurring in a distribution.

[0108] Then the mixed data x a and xb The distance can be calculated as:

[0109] Dis(x a ,x b )=Dis c (x a ,x b )+Dis n (x a ,x b )

[0110] Sort the calculated distance set of all points, and randomly select a line from the 50% of data points with the longest distance in the sorted set as another split point.

[0111] Step 2-2-2: Assignment of points within the cluster.

[0112] In the cluster to be split, calculate the distance between the two split points and the remaining points respectively. The distance calculation method is the same as step 2-2-1, and assign all the remaining points to the closest split point to form two new clusters.

[0113] Step 2-2-3: Statistics of feature information within the cluster. According to the method of step 2-1-1, the numerical and categorical feature information within the cluster is counted.

[0114] Step 2-3: Iteratively assign data points;

[0115] Redistribute the overall data to generate more accurate clusters. The main steps are:

[0116] Step 2-3-1: Distance calculation. Calculate the distribution distance of each data point to the cluster. The more similar the distribution of the point is to the cluster, the greater the probability of belonging to the cluster.

[0117] For categorical features, the calculation method is as follows:

[0118]

[0119] Among them, DL c (x a ,S j ) represents point x a To cluster S j The classification feature distance, Represents data x a The eigenvalues ​​of the categorical columns of the i-th dimension are in cluster S j The probability of occurrence within.

[0120] The distance from the numerical feature to the cluster distribution is calculated as follows:

[0121]

[0122] Among them, DL n (x a ,S j ) represents point x a To cluster S j The numerical feature distance, and Represent cluster S j The mean and standard deviation of ni columns in .

[0123] Then point x a To cluster S j The distance DL(x a ,S j ) is calculated as:

[0124] DL(x a ,S j )=DL c (x a ,S j )+DL n (x a ,S j )

[0125] Step 2-3-2: Point assignment: Reassign each point in the data set to the closest cluster.

[0126] Step 2-3-3: Statistics of feature information within the cluster: According to the method of step 2-1-1, the numerical and categorical feature information within each cluster after allocation is counted.

[0127] Step 2-3-4: Execute steps 2-3-1 to 2-3-3 several times to generate a relatively stable cluster.

[0128] Step 2-4: Calculate the intra-cluster cost;

[0129] For the regenerated clusters, it is necessary to calculate the size of their intra-cluster cost to confirm the next cluster to be split. The intra-cluster cost is defined as the distance from all points in the cluster to the cluster distribution. The distance calculation of each point is the same as step 2-3-1. In addition, the quantity cost factor is added to balance the size of each cluster. The calculation formula of the cluster cost is:

[0130]

[0131]

[0132] Among them, Cost id (S j ) represents cluster S j The quantity cost of |O j | represents cluster S j The amount of data contained in Cost(S j ) represents cluster Sj total cost.

[0133] The total cost of each cluster is calculated according to the above formula, and the cluster with the largest cost is found as the cluster to be split.

[0134] Step 2-5: Iterative splitting;

[0135] Repeat steps 2-2 to 2-4 until the maximum number of splits k is reached. Each split generates a new cluster, generating a total of k+1 clusters.

[0136] Step 2-6: Cluster merging;

[0137] Because unnecessary splits may be performed during the iterative splitting process, that is, the clusters generated after the splitting are relatively similar, after the splitting is completed, if there are similar clusters, they are merged. The specific steps are as follows:

[0138] Step 2-6-1: Calculate the merge cost difference. For any two clusters in the k+1 clusters, calculate their merge cost difference dif ij , calculated as:

[0139] dif ij = Cost(S ij )-Cost(S i )-Cost(S j )

[0140] Among them, S ij For S i and S j Merge the resulting clusters.

[0141] Step 2-6-2: Select the merged cluster. Select the two clusters with the smallest merge cost difference. If the merge cost difference is less than the set threshold, the two clusters will be merged.

[0142] Step 2-6-3: Repeat steps 2-6-1 to 2-6-2 until there are no clusters to merge.

[0143] Step 3. Build a model based on the abnormal feature location algorithm of the isolation forest to identify the abnormal data in each cluster after clustering and locate its specific abnormal features;

[0144] Based on the results of the imbalanced data clustering analysis in step 2, for each cluster after clustering, the abnormal data in the cluster is identified and the abnormal features are located based on the abnormal feature location algorithm based on isolation forest proposed by the present invention, such as Figure 2 As shown, the specific steps include:

[0145] Step 3-1: Feature group division: Obtain the cluster labels and continuous feature data after clustering through step 2, and divide the feature data into different groups according to the cluster labels.

[0146] Step 3-2: Construct an abnormal feature location model based on isolation forest. Based on the feature data obtained after division in step 3-1, an abnormal feature location model based on isolation forest is constructed for each feature group. The main steps are:

[0147] Step 3-2-1: Set model parameters.

[0148] The abnormal feature localization algorithm based on isolation forest proposed in the present invention is based on the classic isolation forest algorithm, but adopts a multi-degree-of-freedom hyperplane segmentation strategy in the construction of isolation trees. It extracts several features from the feature data each time to construct a unit hyperplane to segment the data, instead of the classic isolation forest method of randomly selecting a single feature to segment the data each time. It has stronger robustness and can identify anomalies in associated data.

[0149] Before starting, you need to set the relevant parameters of the positioning model, including the number of isolated trees to be constructed, the number of samples for each isolated tree, the degrees of freedom of the cutting hyperplane, the maximum depth of the isolated tree, etc.

[0150] Step 3-2-2: Sampling.

[0151] According to the sampling number set in step 3-2-1, for each isolated tree, a specified number of samples are randomly selected from the data to be managed using replacement sampling. A root node is set and the root node is used as the current node.

[0152] Step 3-2-3: Feature hyperplane segmentation.

[0153] According to the splitting hyperplane degrees of freedom set in step 3-2-1, several features of the set hyperplane degrees of freedom are randomly selected on the current node as target features for splitting, and the left subtree and right subtree of the node are constructed respectively according to the splitting results; at the same time, the target feature information of the current node is saved.

[0154] Step 3-2-4: Build an isolation tree.

[0155] Iterate step 3-2-3 in the child node until there is only one data in the child node (cannot continue to split) or the child node has reached the maximum depth of the tree set in step 3-2-1, then stop iterating and complete the construction of an isolated tree.

[0156] Step 3-2-5: Extract segmentation path information.

[0157] After completing the construction of an isolated tree through step 3-2-4, the target feature information selected on the entire path from the root node to the leaf node is extracted and saved based on step 3-2-3.

[0158] Step 3-2-6: Build an isolation forest.

[0159] Based on the number of isolated trees to be constructed set in step 3-2-1, steps 3-2-2 to 3-2-5 are executed repeatedly until all isolated trees are constructed and path target feature information is extracted. In view of the low efficiency of sequentially constructing a large number of isolated trees, a multi-threaded parallel tree construction solution is adopted.

[0160] Step 3-3: Calculate the anomaly score.

[0161] Based on the abnormal feature location model constructed in step 3-2, the feature data output in step 3-1 is input into the model. For each feature data, its path depth h(x) in each isolated tree is calculated, that is, the number of edges that the data passes through from the root node of the tree to the leaf node. Then, the average path length of the data in all trees is calculated. The calculation formula is:

[0162]

[0163] Among them, E(h(x)) represents the average path length, T represents the number of trees in the isolation forest, and h t (x) represents the path length of the data point in the tth tree.

[0164] Then, the E(h(x)) of all data points is normalized to obtain the anomaly score of each data point, which is calculated as follows:

[0165]

[0166] Among them, score represents the calculated anomaly score, n represents the amount of data of the input sample when building the tree, that is, the number of samples set in step 3-2-1, and c(n) is the global average path length, which is used for normalization processing. The formula is:

[0167] c(n)=2H(n-1)-2(n-1) / n

[0168] Among them, H(k)=ln(k)+ε, ε=0.5772156649, ε is the Euler constant.

[0169] At the same time, in the process of calculating the abnormal score of each data, the calculation result of the data on each tree is retained and used as the weight score for subsequent positioning of abnormal features. The calculation formula is:

[0170]

[0171] Among them, weight_score represents the weight score.

[0172] Step 3-4: Locate abnormal features.

[0173] Based on the output result of step 3-3, the abnormal features are located.

[0174] Step 3-4-1: Based on step 3-2, obtain the segmentation path information of each data in each tree and the depth of the tree finally reached, and extract the target feature information M of the data at the intermediate nodes of each layer of the path based on the path information.

[0175] Step 3-4-2: Based on step 3-4-1, obtain the target feature information M, and based on the maximum depth h of the tree set in step 3-2-1 max , calculate the limiting depth l d , the calculation formula is:

[0176] l d =floor(0.5*h max )

[0177] Where floor(·) means rounding down.

[0178] The shape of the target feature information M is N×T, where N represents the total amount of data, T represents the number of trees set, and m i,t It represents the target feature information of the path that the i-th data passes through on the t-th tree. The depth of the data on the t-th tree can be expressed as |m i,t |. Now according to the calculated limiting depth l d Divide the target feature information of each data in all trees into two parts, and calculate the depth of the data in each partial tree separately, which is less than the limit depth l d The frequency information of the target features involved, that is, f ano and f nor :

[0179] f ano =freq(m i，t ),|m i，t |≤l d

[0180]

[0181] Among them, freq(·) represents the calculation of m i，t The frequency of each target feature involved in Indicates taking the first l d Deep target feature information.

[0182] Step 3-4-3: Multiply the weight score calculated in step 3-3 by the frequency information calculated in step 3-4-2, and calculate the limit depth l according to step 3-4-2. d Accumulate and average them respectively. The calculation formula for one piece of data is:

[0183]

[0184]

[0185] Among them, T ano Indicates that the depth of a piece of data passing through each tree is less than the limit depth l d The number of trees, T nor Indicates that the depth of a piece of data passing through each tree is greater than the limit depth l d +1 the number of trees. At the same time, according to the limit depth l d The weight score of each data is also divided into two parts, namely weight_score ano and weight_score nor ,but Indicates that in the i-th data, the data path depth is less than the limit depth l d The weight score of the tth tree among all trees of the condition; Indicates that in the i-th data, the data path depth is greater than the limit depth l d The weight score of the tth tree among all trees of the condition.

[0186] Step 3-4-4: Get based on step 3-4-3 and According to the more abnormal characteristics The larger the value, The smaller the value, the more comprehensive score of characteristic abnormality is calculated.

[0187] The calculation formula is as follows:

[0188]

[0189] Among them, attr_score (i) It represents the feature anomaly score of each feature of the i-th data. The score is limited to the range of 0 to 1 through Min-Max normalization. The data in each row are sorted according to the score of each feature. The higher the score, the greater the probability of feature anomaly.

[0190] Step 3-5: Result output: Based on steps 3-2 to 3-4, the abnormal feature positioning results of each group are obtained, and the abnormal labels and positioning results are output according to the set threshold.

[0191] Step 4. Build a data repair model based on heterogeneous variational autoencoder to accurately repair mixed data anomalies and missing data.

[0192] Based on the abnormal labels and abnormal feature positioning results output by steps 3-5, the present invention proposes a data repair model based on heterogeneous variational autoencoders based on deep learning theory, such as Figure 3 As shown in the figure, by analyzing various data structures, different likelihood models are constructed according to their characteristics, mask structures and decomposition decoders are used to process abnormal and missing data, and a hierarchical structure is designed to capture the correlation between features, so as to accurately repair the abnormal and missing mixed data. The specific steps include:

[0193] Step 4-1: Data feature transformation: Based on the original mixed data and data structure information obtained in step 1, the original data is subjected to feature transformation to convert the original data into feature data acceptable to the neural network.

[0194] Based on the data structure information identified in step 1, the classification and numerical features in the original data are obtained. For the classified data, the classification data is encoded by encoding technology, and the encoding technology includes Label-Encoder and One-Hot, the classification data is converted into numerical representation, and the encoding information of the classification data is saved;

[0195] For numerical data, it is divided into three types: real number, positive real number and count (positive integer) data according to the acquired numerical feature information.

[0196] If the feature type is real, the new feature value is equal to the original feature value minus the mean of the feature;

[0197] If the feature type is a positive real number, the new eigenvalue is equal to the original eigenvalue plus 1 to ensure that it is a positive number when the logarithm operation is taken later;

[0198] If the feature type is count (positive integer), the new feature value is equal to the original feature value minus 1 so that the count feature starts at 0.

[0199] Step 4-2: Set the mask structure. According to the results of anomaly detection and abnormal feature location in step 3, mask the feature data identified as abnormal or missing; at the same time, adopt a random masking strategy, set the mask threshold percentage, and randomly mask each feature at a certain ratio. Combine the two masking processes to determine the mask structure that is finally input into the network.

[0200] Step 4-3: Build a data repair model based on heterogeneous variational autoencoders. Based on the mask structure of step 4-2, mask the feature data, and reconstruct the masked data by learning the distribution of non-masked data, so as to achieve the effect of repairing outliers. The specific network structure is set as follows:

[0201] The data restoration model based on heterogeneous variational autoencoder mainly includes two stages: recognition model (RecognitionModel) / encoder (Encoder) and generation model (Generative Model) / decoder (Decoder).

[0202] In the recognition model stage, the encoder aims to obtain the potential representation z of the input data x. In order to enable the network to process data masked by the mask structure, the encoder is modified to focus only on the observed data:

[0203]

[0204] Among them, q(z|x o ) represents the distribution probability function of the real observed data, x o represents the unmasked observation data, represents the input data after the masked data in x is converted to 0 and normalized, N(·) represents Gaussian distribution, μ q and∑ q Distribution representation data The mean and covariance matrix of the encoder can be expressed as:

[0205]

[0206] z=μ+ε*exp(1 / 2*logσ)

[0207] Among them, μ represents the mean, logσ represents the logarithmic standard deviation, enc(·) represents the encoder, and z represents the data input to the decoder. The latent representation of , ε is a value randomly sampled from a Gaussian distribution. Then, the latent variable z generated by the encoder is used as the input of the decoder.

[0208] In the generation model stage, the input latent variable z is mainly decoded and reconstructed through the decoder. In order to be able to handle various types of data, that is, heterogeneous data, the decoder of the traditional VAE model is improved. By decomposing the decoder, different likelihood models are constructed for data of different data types, and an independent DNN network is constructed for each feature according to the data type to generate the parameters of the likelihood model, as follows:

[0209] Construct a Gaussian likelihood model for real-valued data:

[0210]

[0211] Construct a lognormal likelihood model for positive real-valued data:

[0212]

[0213] Construct a Poisson likelihood model for count data:

[0214] p(x nd |γ nd ) = Poisson(x nd |λ d (z n ))

[0215] Construct a multinomial logit model for categorical data:

[0216]

[0217] Among them, γ nd Represents the parameter set of the likelihood model, which is input as the latent variable z n The output of the DNN network, such as μ d (z n ), λ d (z n ),h dv (z n ) and other parameters are stored in γ nd Medium, μ d (z n )and denote the mean and variance respectively, λ d (z n ) represents the parameter average occurrence rate / average number of events of Poisson distribution, h dv (z n ) represents the probability that the d-th dimension feature value of the DNN network output is v. The present invention only lists the likelihood models used by some commonly used data types, and other likelihood models can be constructed for other data types according to requirements.

[0218] Since the repair network constructs an independent likelihood model for each feature according to the data type in the generation model stage, the ability of the generation model to capture the dependency relationship between features is destroyed, and it is impossible to effectively repair abnormal data by relying solely on independent likelihood models. Therefore, the present invention designs a hierarchical structure, which is as follows:

[0219] First, in the recognition model stage, a three-layer network structure is set in the encoder. The first layer Encode to generate latent variable s; convert the category probability of the generated latent variable s into the corresponding one-hot encoding vector and compare it with the input data Merge and use it as the input of the remaining two layers to generate the latent variable z. Then the encoder formula of the recognition model stage can be transformed into:

[0220]

[0221] Where enc_s and enc_z represent the network layers used to generate latent variables s and z in the encoder, respectively, oh represents one-hot encoding, and argmax(·) represents the index position of the maximum value element.

[0222] Then, based on the homogeneous (same type of data) DNN network constructed for each feature in the generative model stage, that is, Y = (y1, y2, ..., y d ), and y d Combined with the latent variable s, an independent likelihood model is input and the likelihood parameters of each feature are output, i.e. (γ1, γ2, …, γ d ), the latent variables s are used to share network parameters in this hierarchical structure, thereby capturing the statistical dependencies between features.

[0223] At the same time, in terms of the loss function, combined with the input mask structure, the loss is calculated only on the observed data, and the evidence lower bound (ELBO) expression is:

[0224]

[0225] Among them, X o represents the unmasked observations in the dataset, Represent the observation and mask parts of the nth data respectively. The first part of the above formula represents the reconstruction loss, which is used to measure the difference between the input value and the reconstructed value; the second part is the KL divergence loss, which is used to measure the difference between the learned distribution and the true distribution.

[0226] To ensure that the model captures all dependencies using latent variables s and z, in the recognition network, the mask features are assumed The posterior distribution of given s and z is consistent with the observed features Conditionally independent, the variational distribution (identification model) can be decomposed into:

[0227]

[0228] By combining the above formula, ELOB can be expressed as:

[0229]

[0230] Specifically, the generative model can be expressed as:

[0231] p(x n ,s n , z n )=p(s n )p(z n / s n )Π dp(x n |γ nd =h d (y nd |s n ))

[0232] Among them, h d (·) represents the DNN network of the d-th dimension data.

[0233] Step 4-4: Abnormal data repair. Based on step 4-3, the network is constructed and trained. Through the training optimization of the recognition model and the generation model in the network, the distribution of each dimension of data generated by the network is continuously close to the real distribution until convergence. Then, the mask data is reconstructed based on the trained model, and the reconstructed value is used as the abnormal repair recommendation value, thereby achieving the purpose of abnormal data repair.

[0234] Step 4-5: Data feature restoration, restore the repair result output from step 4-4 to the format of the original data. First, in step 4-3, the input data was regularized, and the output result needs to be inversely encoded into the format of the network input data; secondly, in step 4-1, different types of data were preprocessed accordingly, and now they are inversely processed and converted into the format of the original data.

[0235] Step 5. Group result output. Based on the output results of steps 2 and 3, run step 4 for each group to obtain the anomaly repair results of each group. Finally, the results of group analysis, anomaly identification and location, and anomaly repair are output separately.

[0236] Method effect display:

[0237] In order to verify the feasibility and effectiveness of the anomaly identification and repair method for imbalanced data proposed in the present invention, relevant experiments were carried out. The amount of experimental data is 110,000. Some of the experimental data are as follows:

[0238] Table 1 Part of experimental data

[0239]

[0240] Based on the above experimental data, the anomaly identification and repair method proposed in the present invention is run. First, the experimental data is analyzed by a hybrid data clustering algorithm based on cost optimization. The clustering results are as follows:

[0241] Table 2 Comparison of clustering results

[0242]

[0243] Table 2 shows some comparison results of two hybrid data clustering algorithms, where “Our” represents the clustering result of the hybrid data clustering algorithm based on cost optimization proposed in the present invention, and “KPT” represents the clustering result of the K-Prototype algorithm.

[0244] According to the clustering results in the table, it can be seen that the results obtained by the clustering algorithm proposed by the present invention are more accurate and reasonable, and each divided cluster has similar characteristics and patterns. For numerical data, namely "YGZDL", "YDRL", "YGZDL" and "ZDF", it can be observed that clusters 1 and 3 have lower numerical levels, while cluster 2 is significantly higher than other clusters, and "YGZDL" and "ZDF" in cluster 4 are at a medium numerical level. For categorical data, it can be observed that in cluster 1, other features have the same pattern, and "DJDM" and "HYLBDM" are significantly different from other clusters; compared with other clusters, "BSFTFSDM" and "BSJFBZ" in cluster 2 are non-zero, indicating that users under this cluster need to share the loss according to the capacity ratio and charge for the shared loss, which is different from other clusters; clusters 3 and 4 are similar, but their "CXDM" is different. Taking into account the numerical and categorical data, it is reasonable and scientific to divide the above data into four clusters.

[0245] In contrast, according to the clustering results of the K-Prototype algorithm, as shown in Table 2, most of the data are classified into cluster 1, and in the overall data, about 98.1% of the data are classified into cluster 1, which shows that the K-Prototype algorithm cannot effectively distinguish data groups when processing mixed attribute data, resulting in most of the data being classified together, resulting in poor clustering effect. Therefore, the unbalanced mixed data clustering algorithm proposed in the present invention is significantly better than the K-Prototype algorithm and can reasonably divide unbalanced mixed data.

[0246] Then, the feature data is divided based on the clustering results, and the abnormal features of each divided group are located by the abnormal feature location algorithm based on the isolation forest; finally, the abnormal data of each group is repaired by the data repair model based on the heterogeneous variational autoencoder. The result example is as follows:

[0247] Table 3. Examples of positioning and repair results

[0248]

[0249] As shown in Table 3 above, there are 4 anomaly examples, where "O" represents the original value of the data, "L" represents the anomaly feature located in the data, "R" represents the anomaly repair value generated by the deep generative model, "cluster" represents the cluster label divided by the hybrid data clustering algorithm based on cost optimization, and "label" represents the anomaly label obtained according to the anomaly threshold. The following is an analysis of the above four records:

[0250] The first record is abnormal in the group with group label 2. The abnormal features located are "BSFTFSDM" and "BSJFBZ", that is, the loss allocation method code and the loss billing flag. The repair result is to repair "BSFTFSDM" from "0" to "2". In actual business, the loss allocation method code and the loss billing flag are closely related. In cluster 2, when the loss billing flag is "calculated [1]", most of the loss allocation method codes are "allocate loss according to capacity ratio [2]", that is, users under this group need to allocate losses according to capacity ratio and be charged for the allocated losses. However, when "BSJFBZ" is "1" in this record, "BSFTFSDM" is "0". Therefore, it is determined that this record is abnormal, and repairing "BSFTFSDM" to "2" meets actual business requirements;

[0251] The second record is abnormal in the group with group label 1. The abnormal feature located is "DLJSFSDM", which is the power calculation method code. The repair result is to repair "DLJSFSDM" from "3" to "1". In group 1, the power calculation method of the remaining records is "1", that is, "fixed ratio", while the power calculation method of this record is "daily agreement", so it is identified as an abnormality and repaired;

[0252] The third record is abnormal in the group with group label 1. The abnormal features located are "XSJFBZ" and "XSJSFSDM", which are the line loss billing flag and the line loss calculation method code. The repair result is to repair "XSJFBZ" from "1" to "0". In actual business, the line loss billing flag and the line loss calculation method are related. When the line loss billing flag is "No [0]", that is, the line loss allocated to the metering point is not charged, the line loss calculation method code should be "Not calculated [0]". However, the line loss billing flag and the line loss calculation method code of this record conflict. Therefore, this record is abnormal in business. Considering the group situation, repairing "XSJFBZ" to "0" meets the actual business requirements;

[0253] The fourth record is abnormal in the group with group label 3. The abnormal feature located is "ZDF", that is, the total electricity fee. The repair result is to repair it from "361.6" to "131.0". In group 3, the mean of the total electricity fee is 24.6 and the median is 15.25. The total electricity fee of this record is significantly higher than that of other records. Therefore, it is judged as an abnormality and provides a repair reference.

[0254] It can be seen from the anomaly identification and repair results of the above experimental data that the method proposed in the present invention can accurately identify and locate anomalies and provide relatively accurate repair reference values.

[0255] At the same time, based on this method, the present invention also provides an abnormality identification and repair system for imbalanced data, such as Figure 5 The system mainly includes data management module, feature processing module, model training module, model management module, feature restoration module and result management module:

[0256] The data management module is used to perform metadata management and data integration management on the data to be governed. It is responsible for organizing the required governance data and initially identifying the data structure based on the metadata management function.

[0257] The feature processing module is used to transform and process the data to be managed and convert it into feature data that can be directly input into various models.

[0258] The operations that can be performed during feature processing include but are not limited to:

[0259] Data structure identification; the data structure identification obtained by the data management module is further refined and identified, including classification feature data structure subdivision and numerical feature data structure subdivision;

[0260] Identify and mark the feature types of input data. Data types are divided into two categories: categorical and numerical. Categorical can be subdivided into classification, ordered, etc., and numerical can be subdivided into integer, real, etc. The type of each feature dimension of the input data is saved according to the specific subdivision type. For categorical features, the number of different values ​​of its corresponding value range needs to be added after each feature. For numerical features, the number of values ​​is 0 by default.

[0261] Categorized data encoding: provides multiple encoding methods for different types of data, including Label-Encoder, One-Hot, etc.

[0262] Numerical data processing: Different methods are used to transform different types of numerical data, including data translation, logarithmic processing, standardization, etc.

[0263] The model training and prediction module is used to manage the model training and prediction process, including three sub-modules: data clustering, abnormal feature location, and abnormal repair.

[0264] The data clustering submodule is used to obtain the processed feature data from the feature processing module, perform clustering analysis on the unbalanced mixed data based on the hybrid clustering algorithm based on cost optimization proposed in the present invention, divide the data to be managed into different cluster groups, and finally output the feature values ​​and cluster group division labels that make the classified data continuous during the clustering process.

[0265] The abnormal feature location submodule is used to obtain continuous feature data and group division labels from the mixed data clustering submodule. Based on the group division labels, for the data of each group, the abnormal feature location algorithm based on isolation forest proposed in the present invention is used to randomly extract a certain number or proportion of data to train an abnormal location model for each group separately, and the feature data of each group is detected based on the trained model, and the final abnormal feature location result is output.

[0266] The anomaly repair submodule is used to obtain the anomaly detection and anomaly location results of each group from the anomaly feature location submodule, and input the original data obtained from the data management module into the data repair model based on the heterogeneous variational autoencoder proposed in the present invention for training, and reconstruct and repair the abnormal data based on the trained model, and output the repair result;

[0267] The model management module is used to manage the models generated by the model training and prediction modules, mainly for model caching and loading, so that the model can be loaded directly from the cache when processing the same table, avoiding the waste of time and computing resources for retraining the model;

[0268] The feature restoration module is used to inversely encode the results output by the model training and prediction modules to convert them into the data structure of the original data. Specifically, the feature processing module performs reverse processing on the original data.

[0269] The result management module is used to manage the results output by the feature restoration module, including the management of mixed clustering, abnormal feature location, and abnormal repair results. It outputs the cluster number and abnormal evaluation value of each data. For data evaluated as abnormal by the method, it outputs the abnormal features (fields) corresponding to the abnormal data records and the values ​​after model repair.

[0270] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for abnormal identification and repair of unbalanced data, used in the field of power systems, characterized in that: The following steps are involved: Step 1. Identify the feature type of the input unbalanced mixed data, the feature type includes: classification type and numerical type; the unbalanced mixed data includes: line loss billing mark, line loss calculation method code, electricity calculation method code, total electricity fee, variable loss allocation method code and variable loss billing mark; Step 2. Construct a model based on the cost-optimized hybrid clustering algorithm, comprehensively consider the differences between the features of the unbalanced mixed data, perform cluster analysis on the unbalanced mixed data, and thus divide the unbalanced mixed data into different clusters; Step 3. Build a model based on the abnormal feature location algorithm of the isolation forest to identify the abnormal data in each cluster after clustering and locate its specific abnormal features; Step 4. Build a data repair model based on heterogeneous variational autoencoder to repair the abnormal and missing mixed data; The step 2 specifically includes: Step 2-1: Data initialization; feature processing and feature information statistics, calculation of feature weights; Step 2-2: Cluster splitting: Split the unbalanced mixed data based on the splitting strategy, split the data into two new clusters, and count the numerical and categorical feature information within the cluster; Step 2-3: Iteratively allocate data points; redistribute the overall data and regenerate clusters; Step 2-4: Calculate the intra-cluster cost; for the regenerated cluster, calculate its intra-cluster cost, which is defined as the distance from all points in the cluster to the cluster distribution; calculate the total cost of each cluster, and find the cluster with the largest cost as the cluster to be split; Step 2-5: Iterative splitting: repeat the above steps until the maximum number of splits is reached, and each split generates a new cluster; Step 2-6: Cluster merging: merging similar clusters; The step 3 comprises: Step 3-1: Feature group division: obtain cluster labels after clustering and continuous feature data, and divide the feature data into different groups according to the cluster labels; Step 3-2: Construct an abnormal feature location model based on isolation forest for each feature group; Step 3-3: Calculate the anomaly score; for each feature data, calculate its path depth in each isolated tree, that is, the number of edges that the feature data passes through from the root node of the tree to the leaf node; then calculate the average path length of the feature data in all trees, normalize the average path length of all data points, and obtain the anomaly score of each data; at the same time, in the process of calculating the anomaly score of each data, retain the calculation result of the data on each tree and keep it as a weight score for subsequent use; Step 3-4: Locate abnormal features; Step 3-5: Result output: Based on steps 3-2 to 3-4, the abnormal feature positioning results of each feature group are obtained, and according to the set threshold, the abnormal label and positioning result are output; The step 4 specifically includes: Step 4-1: Data feature transformation: Based on step 1, the original mixed data and data structure information are obtained, and the feature transformation is performed on the original data to convert the original data into feature data acceptable to the neural network; Based on the data structure information identified in step 1, the classification and numerical features in the original data are obtained. For the classified data, the classification data is encoded by encoding technology, the classification data is converted into numerical representation, and the encoding information of the classification data is saved; Step 4-2: Set the mask structure; according to the abnormal identification and abnormal feature location results of step 3, mask the feature data identified as abnormal or missing; at the same time, adopt a random masking strategy, set the mask threshold percentage, and randomly mask each feature at a certain ratio. Combining the two masking processes, determine the mask structure that is finally input into the network; Step 4-3: Build a data repair model based on heterogeneous variational autoencoder; based on the mask structure of step 4-2, mask the feature data, reconstruct the masked data by learning the distribution of non-masked data, so as to achieve the effect of repairing outliers; Step 4-4: abnormal data repair; based on step 4-3, the network is constructed and trained. Through the training optimization of the recognition model and the generation model in the network, the distribution of each dimension data generated by the network is continuously close to the real distribution until convergence; then, the mask data is reconstructed based on the trained model, and the reconstructed value is used as the abnormal repair recommendation value, so as to achieve the purpose of abnormal data repair; Step 4-5: Data feature restoration, restore the repair result output from step 4-4 to the format of the original data; first, in step 4-3, the input data is regularized, and the output result needs to be inversely encoded into the format of the network input data; secondly, in step 4-1, different types of data are preprocessed accordingly, and then inversely processed to convert them into the format of the original data.

2. The method for identifying and repairing anomalies in unbalanced data according to claim 1, characterized in that: In step 3-2, an abnormal feature location model based on isolation forest is constructed, which specifically includes: Step 3-2-1: Setting model parameters, including the number of isolated trees to be constructed, the number of samples for each isolated tree, the degree of freedom of the segmentation hyperplane, and the maximum depth of the isolated tree; Step 3-2-2: According to the set sampling number, for each isolated tree, randomly select a specified number of samples from the data to be managed by replacing the sampling method, set a root node, and use the root node as the current node; Step 3-2-3: According to the set splitting hyperplane degrees of freedom, randomly select several features of the set hyperplane degrees of freedom on the current node as target features for splitting, and construct the left subtree and right subtree of the node respectively according to the splitting results; at the same time, save the target feature information of the current node; Step 3-2-4: Construct an isolation tree; Iterate step 3-2-3 in the child node until the child node cannot be further split or the child node has reached the maximum depth of the tree set in step 3-2-1, then stop iterating and complete the construction of an isolated tree; Step 3-2-5: Extract segmentation path information; Based on step 3-2-3, extract and save the target feature information selected on the entire path from the root node to the leaf node; Step 3-2-6: Construct an isolation forest; Based on the number of isolated trees required to be constructed set in step 3-2-1, steps 3-2-2 to 3-2-5 are executed in a loop until the construction of all isolated trees and the extraction of path target feature information are completed.

3. The method for identifying and repairing anomalies in unbalanced data according to claim 2, characterized in that: The positioning of abnormal features in step 3-4 specifically includes: Step 3-4-1: Based on step 3-2, obtain the segmentation path information of each data in each tree and the depth of the tree finally reached, and extract the target feature information of the data in the middle node of each layer of the path according to the path information; Step 3-4-2: Obtain target feature information based on step 3-4-1, and calculate the limit depth based on the maximum depth of the tree set in step 3-2-1; and divide the target feature information of each data on all trees into two parts according to the calculated limit depth, and calculate the frequency information of the target features involved in each part of the tree where the depth is less than the limit depth; Step 3-4-3: Multiply the weight score calculated in step 3-3 by the frequency information calculated in step 3-4-2, and add them up according to the limit depth in 3-4-2 and take the average. The weight score of each piece of data needs to be divided into two parts according to the limit depth. Step 3-4-4: Calculate the comprehensive score of characteristic abnormalities.

4. A system for identifying and repairing anomalies in unbalanced data applied to the method for identifying and repairing anomalies in unbalanced data according to any one of claims 1 to 3, characterized in that: include: Data management module, feature processing module, model training module, model management module, feature restoration module, result management module; The data management module is used to perform metadata management and data integration management on the data to be governed. It is responsible for organizing the required governance data and initially identifying the data structure based on the metadata management function. The feature processing module is used to transform the data to be managed and convert it into feature data that can be directly input into various models; The model training module is used to manage the model training and prediction process, including three sub-modules: data clustering, abnormal feature location, and abnormal repair; The data clustering submodule is used to obtain the processed feature data from the feature processing module, perform clustering analysis on the unbalanced mixed data based on the cost-optimized hybrid clustering algorithm, divide the data to be managed into different cluster groups, and finally output the feature values ​​and cluster group division labels that make the classified data continuous during the clustering process; The abnormal feature location submodule obtains continuous feature data and group division labels from the mixed data clustering submodule; based on the group division labels, for each group of data, a certain number or proportion of data is randomly selected using the abnormal feature location algorithm based on the isolation forest to train an abnormal location model for each group, and the feature data of each group is tested based on the trained model to output the final abnormal feature location result; The anomaly repair submodule obtains the anomaly recognition and anomaly location results of each group from the anomaly feature location submodule, and trains the data repair model based on the heterogeneous variational autoencoder based on the original data input obtained from the data management module, reconstructs and repairs the abnormal data based on the trained model, and outputs the repair results; The model management module is used to manage the models generated by the model training and prediction modules, and to cache and load the models, so that the models can be loaded directly from the cache when processing the same table; The feature restoration module is used to reverse the results output by the model training and prediction modules to convert them into the data structure of the original data, specifically, to reverse the processing process of the original data according to the feature processing module; The result management module is used to manage the results output by the feature restoration module, including the management of mixed clustering, abnormal feature location, and abnormal repair results. It outputs the cluster number and abnormal evaluation value of each data. For data evaluated as abnormal, it outputs the abnormal features corresponding to the abnormal data record and the value after model repair.

Images