Anomaly detection methods and systems with propagation and learning capabilities for data governance

Abstract

This invention discloses an anomaly identification method and system with propagation learning capabilities for data governance. The method includes: loading a configuration file, reading the data to be detected, and distinguishing it into numerical fields and categorical fields; preprocessing the data to be detected to obtain sampling intervals and initial labels; sampling data according to the sampling intervals, and training a field-level anomaly classifier and a record-level anomaly classifier by combining interactive labeling operations; using the trained field-level and record-level anomaly classifiers to perform field-level and record-level anomaly identification on the data to be detected, respectively, and outputting the anomaly data. This invention extracts representative data based on a sampling algorithm, labels the data through a user interaction module integrated into the system, expands the number of labels through label propagation, significantly reducing the amount of labeled data required, trains a supervised classification model, and combines active learning to improve the accuracy of anomaly identification for mixed data.

Description

Technical Field

[0001] This invention belongs to the field of data governance and relates to anomaly data identification technology, specifically an anomaly identification method and system with propagation and learning capabilities for data governance. Background Technology

[0002] Data governance is a key approach to realizing the value of data elements, and the identification of outliers is a crucial step in this field. Outliers are typically defined as outliers, noise, etc. (such as observations with significant discrepancies). Outliers can originate from multiple factors, including mechanical failures, instrument errors, and human error, and their generation is usually unavoidable. The presence of these outliers degrades data quality, adversely affects the results of data analysis, and may even lead to serious consequences. Therefore, it is essential to identify and eliminate outliers during the data governance process.

[0003] The commonly used anomaly detection technologies currently include the following:

[0004] Methods based on manual rules and statistics: Manual rule-based methods primarily rely on expert experience to establish rules for data verification; deviations from these rules are considered anomalies. However, rule-based detection methods can only detect anomalies within the established rules, limiting their scope. Statistical anomaly detection methods generally establish a data distribution model, calculate the probability of an object conforming to the model, and mark low-probability objects as anomalies. Some commonly used methods include box plots, the 3σ criterion, and the Grubbs test. These methods all assume that the data follows a certain distribution and use the data for parameter estimation. Other methods include Gaussian mixture modeling and Markov-based models. While statistical methods have good robustness and are suitable for low-dimensional data, they have limitations when dealing with high-dimensional data and are also affected by data distribution and model parameters.

[0005] Supervised and unsupervised machine learning methods: Supervised machine learning assumes that a classifier capable of distinguishing between normal and abnormal classes can be learned in a given feature space. It learns a model from a set of labeled data and then uses the learned model to classify the data to be tested. Common supervised methods include Bayesian networks, support vector machines, and random forests. Unsupervised machine learning includes density detection, special classification methods, and others. Density detection methods estimate the neighborhood density of each data instance; observations located in low-density neighborhoods are considered anomalous. Classic density methods include DBSCAN and the Local Anomaly Factor (LOF).

[0006] Deep learning-based methods: Deep learning techniques can automatically learn high-level representations and features of data, effectively capturing complex patterns and anomalous behaviors. Common methods include autoencoders (AEs), variational autoencoders (VAEs), and generative adversarial networks (GANs). An autoencoder consists of an encoder and a decoder, learning the latent structure of the data by compressing the input data into a low-dimensional encoded representation and reconstructing the original data. A generative adversarial network is an adversarial model composed of a generator and a discriminator. In anomaly detection, a generator can be used to generate a normal data distribution, and a discriminator can be used to distinguish generated samples from real samples.

[0007] Supervised anomaly detection techniques outperform unsupervised anomaly detection techniques because supervised methods are trained using labeled samples. However, acquiring labeled data is extremely labor-intensive, making large-scale industrial applications difficult. Unsupervised anomaly detection methods can be trained without labels, but due to the lack of label information, data processing is more subjective, relying more on the distribution and statistical characteristics of the data. Furthermore, different algorithms and parameter settings can lead to different anomaly detection results, making the interpretation and comparison of results difficult.

[0008] With the advent of the data era, where data has become a factor of production, the realization of data value places higher demands on data quality. Data governance has become a key approach to improving data quality, and how to efficiently and accurately identify anomalous data in mixed data has become one of the core issues in structured data governance. However, existing anomaly detection algorithms have many shortcomings in handling mixed data, and the target scale for detection is relatively singular. Mixed data refers to data containing both numerical and categorical features. Methods based on manual rules and statistics are often limited to categorical data; traditional unsupervised learning methods have too low anomaly detection accuracy; the classic IF algorithm only focuses on anomaly detection in numerical data and is not effective for mixed data. Deep learning and other methods rely on data reconstruction to identify anomalies, which often requires clean training data; supervised methods require complete normal / anomaly labeling of the training set beforehand. These prerequisites are often not met in practice, and labeling massive amounts of data requires a huge investment of manpower. More importantly, all existing methods assume that different data attributes are independent, which is completely unrealistic. In reality, different data attributes are often correlated; in addition, existing methods can often only identify whether there are anomalies in the entire record, but cannot identify the specific fields where anomalies exist. Summary of the Invention

[0009] Purpose of the invention: To overcome the shortcomings of existing technologies in identifying different types of anomalies in mixed data, this invention provides a semi-supervised anomaly identification method and system with propagation learning capabilities for data governance. First, an anomaly feature matrix is ​​constructed based on an integrated unsupervised anomaly detection algorithm. Then, representative data is extracted using a sampling algorithm. The data is labeled through a user interaction module integrated into the system. The number of labels is expanded through label propagation, significantly reducing the amount of labeled data required. By training a supervised anomaly detection model and combining it with active learning, the identification accuracy is improved. This method can not only identify whether the current record is abnormal, but also identify specific abnormal fields.

[0010] Technical Solution: To achieve the above objectives, this invention provides an anomaly identification method with propagation and learning capabilities for data governance, comprising the following steps:

[0011] S1: Load the configuration file, read the data to be tested from the data source, and distinguish the data to be tested into numeric fields and categorical fields;

[0012] S2: Preprocess the data to be detected to obtain the sampling range and initial label;

[0013] S3: Sampling data based on the sampling interval, combined with interactive labeling operations, to train field-level anomaly classifiers and record-level anomaly classifiers;

[0014] S4: Use the trained field-level anomaly classifier and record-level anomaly classifier to perform field-level anomaly identification and record-level anomaly identification on the data to be detected, and output the anomaly data.

[0015] Further, the configuration file loading in step S1 specifically involves: obtaining parameter settings and normal data information from the configuration file. Parameter settings include the data path to be detected, the number of samples, the feature generation algorithm, and the label propagation method. Normal data information refers to the normal records and normal category values ​​of the categorization fields known to the user on the data. The record set Lr and the field set Lc are initialized, and the normal data information is stored in the corresponding structure, or empty if there is no such structure, such as Lr = {x1, x2, ...}, Lc = {j1: (c1, c2, ...), ...}, where x i It is a complete record, where c1 represents the category value in column j1.

[0016] Furthermore, in step S1, data reading involves: reading the data to be tested D from the data source, distinguishing the column names of numeric fields and categorical fields in the data, and storing the data and column names in the corresponding structures.

[0017] Furthermore, the preprocessing in step S2 specifically includes the following steps:

[0018] A1: Generate feature set

[0019] A1-1: For numerical fields, configure parameters according to the feature generation algorithm and generate features; use one or more of the following algorithms to generate features: Isolation Forest (IF), Empirical-Cumulative-distribution-based Outlier Detection (ECOD), and Histogram-based Outlier Score (HBOS). The method for generating features is to detect each column of numerical fields, use the outlier score of the algorithm as the feature of the field, generate a total of m-dimensional features, and then use Min-Max standardization for the features, where m is the number of times the above algorithms are used.

[0020] A1-2: For categorical fields, use the frequency transformation method of homogeneity to generate a one-dimensional feature for each field;

[0021] A1-3: Store the features corresponding to each field in a collection;

[0022] A2: Group identification and coding, including the following steps:

[0023] A2-1: Based on clustering methods, the features of numerical fields are converted into encoded representations of clustering results. First, a clustering model is trained for each numerical column, and the K-Means algorithm is used for clustering. After constructing the clustering model, each numerical value is replaced with its corresponding cluster label. Let the j-th column generate four clusters C. j ={c0,c1,c2,c3}, replace all data in this column with the cluster number {0,1,2,3} to which it belongs;

[0024] A2-2: Convert the categorical field features into coded representations; for any categorical field column, sort it according to the magnitude of its frequency features and convert it into a coded representation. For example, the category with the highest frequency in the column is coded as 0, the next highest as 1, and so on. After converting all columns, generate a coded data table DC.

[0025] A3: Obtain the sampling interval and initial label, including the following steps:

[0026] A3-1: Pre-scoring using Enhanced Isolated Forest:

[0027] To alleviate the user's interaction burden, it is necessary to minimize the amount of data to be labeled by the user. An unsupervised algorithm is used to pre-evaluate abnormal records, identifying clearly normal and abnormal data. A portion of the data with greater uncertainty is sampled and then manually labeled, improving the effectiveness of the sampling. The augmented isolated forest algorithm model provided in this invention can effectively identify mixed-type data anomalies. It calculates an anomaly score for each data point and sorts the scores from highest to lowest; a higher score indicates a more abnormal data point.

[0028] A3-2: Segment the data and determine the sampling interval

[0029] Based on a set threshold (usually set to 1% to 5%), the data is split according to the score of the enhanced isolated forest. The group labels near the threshold are the most difficult to distinguish. After splitting, the abnormal data set A and the normal data set N are obtained.

[0030] Remove duplicates from sets A and N to obtain duplicate sets A' and N'. Then, further divide sets A' and N' into 50% segments, denoted as AC, AI, NI, and NC. The latter 50% of set A' and the former 50% of set N', i.e., AI and NI, are considered to be uncertain sets because this part of the data is closer to the outlier threshold. These segments are divided into sampling intervals, while samples outside the sampling intervals, i.e., AC and NC, are considered to be deterministic samples.

[0031] A3-3: Obtaining Initial Tags

[0032] Data outside the sampling interval are labeled based on the enhanced isolated forest score to obtain initial labels. The overall labeling method is as follows:

[0033]

[0034] Where, d i It is the i-th data point in the original dataset D, l i It is d i The initial tag.

[0035] Furthermore, data standardization in step A1 is the process of unifying data with different dimensions to the same dimension. Its purpose is to eliminate the influence of different dimensions between variables, making different indicators comparable. Specifically, Min-Max standardization in step A1 involves:

[0036] The formula for linearly mapping the original data to the range [0,1] is as follows:

[0037]

[0038] Where min(X) and max(X) are the maximum and minimum values ​​of the data in this column, respectively;

[0039] The IF algorithm is an unsupervised machine learning algorithm mainly used to detect anomalies in numerical data. It finds outliers in the data by constructing a set of isolation trees and calculates the anomaly score of the sample based on the path length of the isolation trees.

[0040] The ECOD algorithm, also an unsupervised machine learning algorithm, first estimates the latent distribution of the data in a nonparametric manner by calculating the empirical cumulative distribution function (ECDF). Then, it uses these empirical distributions to estimate the tail probability of each data point. Finally, it calculates the outlier value of each data point by aggregating the estimated tail probabilities. The method for calculating the ECDF is as follows:

[0041]

[0042]

[0043] in, and Let be the left-tailed ECDF and right-tailed ECDF in the j-th dimension, respectively. Θ is an indicator function, which is 1 when its parameter is true and 0 otherwise. The outlier calculation method for each sample is as follows:

[0044]

[0045]

[0046]

[0047] Among them, O left (x), O right (x), O auto (x) represents the left-tailed outlier, right-tailed outlier, and corrected outlier, respectively, where d is the dimension of x and γ is the skewness of the data.

[0048] The final anomaly score for the sample is:

[0049] O(x) = max{O left (x),O right (x),O auto (x)}

[0050] The HBOS algorithm is as follows: Histogram detection HBOS is a statistical method. First, the column is normalized using the Min-Max method to scale it to the same scale. Second, the data range is divided into several intervals, the number of samples in each interval is counted, and the relative frequency of the samples in that interval is calculated. Assuming the column has k intervals, represented as B = {b1, b2, ..., b...} k}, then the relative frequency corresponding to the i-th interval is:

[0051]

[0052] Where n(i) represents the interval b i The number of samples in the sample, where n represents the total number of samples.

[0053] For each sample, map it to the corresponding interval. Suppose sample x falls in interval b. i In the meantime, the calculated anomaly score s(x) is:

[0054] s(x)=-log(p i )

[0055] Furthermore, the frequency conversion and frequency differentiation method in step A1 is specifically as follows:

[0056] The frequency conversion calculates the occurrence frequency of each category in the field and converts it into a frequency representation. The conversion method is as follows:

[0057]

[0058] Where c(i) is the i-th category in this field, and n is the total number;

[0059] Since categorized data often contain elements with the same frequency, making them indistinguishable, this invention proposes a frequency-differentiation method. This method distinguishes categories with the same frequency by adding random values, minimizing the impact on the results. First, all category frequencies are sorted in ascending order. Then, random values ​​are added to values ​​with the same frequency, ensuring that the frequency order remains unchanged after adding the random values. Let the sorted set be P = {p1, p2, ..., p...}. l}, then the random value r is selected in the following way:

[0060] r = random(p) i -p left ,p right -p i )

[0061] Where, p left It is p i The left side is not equal to p i The maximum value of p right It is p i The right side is not equal to p i The minimum value.

[0062] Furthermore, the specific process of clustering using the K-Means algorithm in step A2-1 is as follows:

[0063] B1: Arbitrarily initialize k sample points from all sample points as cluster centroids;

[0064] B2: Calculate the distance between all points and the k centroids, and assign each data point to the cluster containing the nearest centroid;

[0065] B3: Recalculate the centroid of each cluster based on the newly divided points;

[0066] B4: Repeat steps B1 to B3 until the centroid converges;

[0067] The method for calculating the centroid is as follows:

[0068]

[0069] Where, μ i Let represent the i-th centroid, and ni be the number of contained points of cluster Si.

[0070] Furthermore, in step A3-1, addressing the issue that the classic IF algorithm is only applicable to numerical data and cannot handle categorical or mixed data, this invention proposes an enhanced isolated forest algorithm, which can effectively detect anomalies in mixed data and has strong robustness, used for anomaly scoring of mixed data.

[0071] The specific computational steps of the augmented isolated forest algorithm are as follows:

[0072] C1: Calculate the information gain ratio of the categorical fields.

[0073] Information gain is used to measure the degree of influence of a field on the classification result. The information gain ratio is the information gain divided by the entropy of the dataset after partitioning. This can penalize features with a large number of values. Its calculation method is as follows:

[0074]

[0075]

[0076]

[0077] in, and `<information_gain_ratio>` and `<information_gain_growth_ratio>` represent two different categorical fields, respectively. `<information_gain_ratio>` and `<information_gain_growth_ratio>` represent the calculation of the information gain ratio and information gain between the categorical fields, respectively. Represents calculated fields right Information gain ratio; H(·) and H(·|·) represent the calculation of information entropy and conditional entropy, respectively; v m Represents category field The m-th attribute value in p(v) m ) indicates in the field In the middle, the calculation equals v m The ratio of the number of attribute values ​​to the total length of the field;

[0078] C2: Calculate correlation

[0079] Because information gain ratio is asymmetric and cannot effectively measure the correlation between categorical fields, it is necessary to transform the information gain ratio to calculate the correlation between categorical attributes; as shown below:

[0080]

[0081] in, Represents categorized fields and The correlation between them;

[0082] C3: Feature Processing

[0083] Based on the correlation analysis results between the categorical fields in the data, significantly related categorical fields are concatenated to generate new categorical fields, and the original categorical fields are deleted, thus forming new mixed data. Features are extracted from the existing categorical fields, that is, the frequency transformation method of homogeneity is used to convert the categorical data into numerical data, enabling it to identify the mixed data;

[0084] C4: Sampling

[0085] A certain number of subsamples are randomly selected from the data; for each isolated tree, a root node is set and the root node is used as the current node;

[0086] C5: Attribute Splitting

[0087] At the current node, several dimensions are randomly selected as target dimensions for segmentation, and the left and right subtrees of the node are constructed based on the segmentation results. The segmentation strategy is as follows:

[0088]

[0089] in, This represents a vector composed of several target dimensions. The intercept vector is derived by taking values ​​from a uniform distribution between the maximum and minimum values ​​in each target dimension. The normal vector is generated randomly for each target dimension using a standard normal distribution.

[0090] Through calculation Obtain the splitting hyperplane. If it is less than 0, that is, it is located below the splitting hyperplane. The corresponding sample falls into the left subtree, and the rest fall into the right subtree.

[0091] C6: Constructing an Enhanced Isolated Tree

[0092] Repeat step C5 in the child nodes until there is only one piece of data in the child node or the child node has reached the maximum depth of the set tree, then stop the step and complete the construction of the isolated tree;

[0093] C7: Building Enhanced Isolated Forests

[0094] Based on steps C4 to C6 above, construct multiple reinforced isolated trees to form an reinforced isolated forest;

[0095] C8: Calculate anomaly scores

[0096] Input the dataset processed in step C3 into the augmented isolation forest. For each data point, calculate its average path length in the augmented isolation forest using the following formula:

[0097]

[0098] Where E(h(x)) represents the average path length, T represents the number of trees in the augmented isolated forest, and h t (x) represents the path length of the data point on tree t;

[0099] By normalizing the average path length of all data, an anomaly score can be obtained for each data point. The calculation formula is as follows:

[0100]

[0101] c(n) = 2H(n-1) - 2(n-1) / n

[0102] H(k)=ln(k)+ε

[0103] Where n represents the amount of data in the input samples for building the tree, c(n) is the global average path length used for normalization, and ε is Euler's constant, ε = 0.5772156649.

[0104] Furthermore, step S3 specifically includes the following steps:

[0105] D1: Data sampling and result acquisition, including the following steps:

[0106] D1-1: Sample the data within the sampling interval.

[0107] Since each column in different records may contain the same error, in the data preprocessing module, each data column is divided into several data clusters and represented by an encoding. To minimize the number of labels required, ideally, the sampled data should cover all unlabeled clusters in each column. Therefore, sampling is performed through multiple iterations. In each iteration, the sampling probability of each data point is calculated using the Softmax function.

[0108]

[0109]

[0110] Where, p i It is d i The sampling probability, p ij It is data d i The probability of in the j-th column, assuming d i The element in column j belongs to cluster c, N jc represents the number of samples that have been taken from the c-th cluster in the j-th column, and k is the total number of clusters in the j-th column. Using this probability is beneficial for obtaining data containing clusters that have been sampled less frequently. Sampling is performed according to the sampling probability until the number of samples reaches a preset value, forming a sample set S.

[0111] D1-2: Use the quartile method to filter the categorical data to be labeled.

[0112] For each categorical field, the quartile values ​​are calculated based on frequency characteristics, and categories with frequencies below the lower quartile threshold are added to the categorical field sampling set S. l The lower quartile boundary is calculated as follows:

[0113] IQR = 1.5 * (Q3 - Q1)

[0114] b = Q1 - IQR

[0115] Q1 and Q3 are the 25th percentile and 75th percentile, respectively;

[0116] D1-3: Obtaining Tagging Results Using the User Interaction Module

[0117] Based on the user interaction module, the sampling set S and the classification field sampling set S l The data is then passed to the user for annotation, and the annotation results are parsed to obtain data labels.

[0118] D2: Train a record-level classifier through interactive active learning, including the following steps:

[0119] D2-1: Training the initial classification model

[0120] Compared with traditional decision tree models, the LightGBM model selected in this invention has higher training speed and accuracy. LightGBM is selected as the classifier, and combined with the acquired data labels, including the record set Lr generated by the data and configuration loading module, the initial labels in step S2 of the data preprocessing module, and the data labels obtained by the interaction module, the corresponding data in the encoding table DC is used as input to obtain the initially trained model.

[0121] D2-2: Selecting samples to be labeled based on predicted probabilities

[0122] The model is used to predict the deduplication sets A' and N', and the predicted probabilities of each sample belonging to the normal and abnormal classes are obtained. The calculation formula is as follows:

[0123]

[0124] Where P(y=k|x) represents the probability that the sample belongs to class k, and the score k The score represents the average score of category k across all decision trees. k The calculation method is as follows:

[0125]

[0126]

[0127] Where pi(k) represents the probability that a sample is assigned to a leaf node in the i-th tree and that sample belongs to class k at that leaf node, and N is the number of decision trees, n i This is the total number of nodes in this leaf node, n. i (k) is the number of leaf nodes belonging to category k;

[0128] The uncertainty probability of each sample is calculated based on the predicted probability, as follows:

[0129] prop=Abs(P(y=0|x)-0.5)+Abs(P(y=1|x)-0.5)

[0130] Where P(y=0|x) and P(y=1|x) represent the probabilities of the sample being normal and abnormal, respectively, and Abs() represents the absolute value operation;

[0131] Sort all predicted samples according to their uncertainty probability, take the 10 samples with the highest uncertainty, and update the sample set S accordingly;

[0132] D2-3: Labeled Samples

[0133] The samples obtained in the previous step are passed into the user interaction module to obtain the labels and update the label set.

[0134] D2-4: Update the model

[0135] Train a new model using all the data with existing labels;

[0136] D2-5: Repeat steps D2-2 to D2-4 until the number of samples in step D2-2 reaches the preset value, and obtain the final record-level anomaly detection classifier m0.

[0137] D3: Field Tag Propagation

[0138] To maximize the use of known information and reduce the number of labels required, this invention designs a field label propagation scheme based on encoding results. For numerical columns, existing labels are propagated to other data in the same cluster according to the clustering results to obtain additional labeled field data; for categorical fields, labels are propagated to other data with the same attribute value.

[0139] Since a cluster may have multiple normal or abnormal labels with contradictory meanings, this invention designs two propagation methods—homogeneous propagation and majority propagation. Homogeneous propagation only propagates to clusters that do not contain contradictory labels, that is, clusters containing only one type of label. Assume that for the j-th column field, there are r clusters C = {c1, c2, ..., c...} r If c i If all the labeled data labels in c are 1, then i Other data labels in the data were also propagated as 1;

[0140] Most propagation will spread user tags in clusters with mixed tags, choosing the tag with the greater prevalence in the cluster for propagation. If c i If the proportion of labeled data in the data is larger, then c i Other data labels in the data are also propagated as 1; if the proportions are the same, they are not propagated.

[0141] Based on the configuration file settings, select a propagation method and propagate the labels for each column field separately to obtain the propagated field-level labels.

[0142] D4: Training field-level classifier

[0143] For each numerical field j, filter the data with existing labels and train a LightGBM classifier m using the features generated in step S2. j We obtain l field-level anomaly classifiers M = {m1, m2, ..., m}. l}, where l represents the number of columns containing numeric fields.

[0144] Furthermore, in step D2-1, LightGBM is an algorithm model based on gradient boosting decision trees, applicable to classification and regression tasks. It employs a gradient boosting method, iteratively improving the model's accuracy. The LightGBM model operates as follows: initially, the model is a simple decision tree; then, in each iteration, more decision trees are gradually added to correct the previous model's prediction errors; a histogram algorithm is used to optimize the training process of the decision trees, discretizing the values ​​of continuous features and dividing the dataset into multiple histograms; within each histogram, the gradient and number of samples within that histogram are calculated, and then the optimal split point is selected; the gradient is calculated as follows:

[0145]

[0146] in It is the loss function, where y is the true value. These are model predictions.

[0147] This invention also provides an anomaly detection system for data governance with propagation and learning capabilities, including a data and configuration loading module, an algorithm library integration module, a data preprocessing module, a user interaction module, a model training module, and an anomaly detection module;

[0148] The data and configuration loading module is used to perform the loading of configuration files and loading of data in step S1;

[0149] The algorithm library integration module is used to store various algorithms required by other modules for easy access at any time;

[0150] The data preprocessing module is used to execute step S2, which mainly realizes the functions of cleaning and transforming data, pre-evaluation, etc., so as to facilitate subsequent abnormal model training and identification, including three steps: generating feature set, group identification and coding, and obtaining sampling interval and initial label.

[0151] The user interaction module is used to receive sampling result data from other modules of the system, display the sampling results to the user and perform annotation operations, and finally receive the user's annotation results and save them in the corresponding structure.

[0152] After a sampling process, users need to label each record with field and record anomalies. Users label one record at a time, using a label format like {0,1,0,0,...,1}, where the last value is the record-level anomaly label, and the first d values ​​represent the anomaly label for each field, where d is the total number of columns in the data. 0 represents normal, and 1 represents anomaly. If the output data is a single field value to be labeled, it is labeled directly. By parsing the user input, record labels and field labels are stored in record label sets and field label sets, respectively. The process proceeds to the next stage only after all user labeling is completed and the parsing results are correct; otherwise, the labeling stage restarts.

[0153] The model training module and the user interaction module work together to execute step S3, obtain sampled data and labeling results, and train field-level anomaly classifiers and record-level anomaly classifiers.

[0154] The anomaly identification module is used to execute step S4, which identifies record anomalies and field anomalies based on the trained record-level anomaly classifier and field-level classifier, and finally outputs the anomaly data.

[0155] The anomaly detection module's detection process is as follows: First, unlabeled record data is detected by inputting the encoding table DC into the trained record-level classifier to identify record-level anomalies; second, the numerical column features are input into the corresponding column's field-level classifier to detect numerical column field anomalies; finally, based on the field set Lc generated in the configuration loading module and the user's annotation results, combined with label propagation, the anomalies of most data in the classification column can be determined, and the frequency features of the remaining data are all within the calculated quartile limits, and are considered to belong to the normal category.

[0156] This invention designs a semi-supervised method. Compared to supervised learning methods that rely entirely on labeled data, semi-supervised learning algorithms can train models using a limited amount of labeled data, combined with other unlabeled data. Simultaneously, it leverages the information from labeled data to reduce the impact of noise, thereby improving the model's robustness and stability. The semi-supervised method designed in this invention first extracts representative data using a sampling algorithm, then labels the data through a user interaction module integrated into the system and expands the number of labels through label propagation. This significantly reduces the amount of labeled data required, which is then used to train the classification model and combined with active learning to improve recognition accuracy.

[0157] To improve the detection effect of mixed data, this invention applies different feature processing and detection methods to categorical fields and numerical fields in the system. In order to identify the various anomaly types contained in the data, anomaly detection classifiers of multiple scales are set in the system, including field-level classifiers and record-level classifiers, to deal with outliers within each field and correlation anomalies between fields, respectively.

[0158] Beneficial Effects: Compared with existing technologies, this invention enables the processing of mixed data and anomaly detection by applying different preprocessing methods to numerical and categorical fields. Through clustering and label propagation methods, a classifier can be trained with a small number of user labels, forming a semi-supervised detection model that significantly reduces the workload of manual labeling. Furthermore, active learning further improves detection accuracy. By integrating multiple configurations of anomaly detection classifiers into the method, it can simultaneously detect various types of anomalies in mixed data and discover potential anomaly patterns in the dataset. This is of great significance for various business operations in actual production, such as power consumption verification and fraud detection. Attached Figure Description

[0159] Figure 1 This is a design drawing of the system of the present invention;

[0160] Figure 2 This is a partial data diagram from this embodiment;

[0161] Figure 3 This is a partial configuration diagram of this embodiment;

[0162] Figure 4 This is a diagram illustrating the feature set in this embodiment;

[0163] Figure 5 This is a diagram illustrating the encoding conversion table in this embodiment;

[0164] Figure 6 This is a diagram showing the pre-assessment ranking results of the enhanced isolated forest in this embodiment;

[0165] Figure 7 This is a diagram of the marking process in this embodiment;

[0166] Figure 8 This is a diagram illustrating the changes in the active learning results in this embodiment;

[0167] Figure 9 This diagram illustrates the tag propagation process in this embodiment. Detailed Implementation

[0168] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0169] like Figure 1As shown, the present invention also provides an anomaly detection system with propagation learning capability for data governance, including a data and configuration loading module, an algorithm library integration module, a data preprocessing module, a user interaction module, a model training module, and an anomaly detection module;

[0170] The data and configuration loading module is used to load configuration files and data;

[0171] The algorithm library integration module is used to store various algorithms required by other modules for easy access. The algorithms include Min-Max normalization, IF, ECOD, HBOS, homogeneity, and enhanced isolated forest, among which homogeneity and enhanced isolated forest are the key improved algorithms of this invention.

[0172] The data preprocessing module mainly performs functions such as data cleaning, transformation, and pre-evaluation to facilitate subsequent anomaly model training and identification. It includes three steps: generating feature sets, group identification and coding, and obtaining sampling intervals and initial labels.

[0173] The user interaction module is used to receive sampling result data from other modules of the system, display the sampling results to the user and perform annotation operations, and finally receive the user's annotation results and save them in the corresponding structure.

[0174] After a sampling process, users need to label each record with field and record anomalies. Users label one record at a time, using a label format like {0,1,0,0,...,1}, where the last value is the record-level anomaly label, and the first d values ​​represent the anomaly label for each field, where d is the total number of columns in the data. 0 represents normal, and 1 represents anomaly. If the output data is a single field value to be labeled, it is labeled directly. By parsing the user input, record labels and field labels are stored in record label sets and field label sets, respectively. The process proceeds to the next stage only after all user labeling is completed and the parsing results are correct; otherwise, the labeling stage restarts.

[0175] The model training module works in conjunction with the user interaction module to acquire sampled data and labeling results, and to train field-level anomaly classifiers and record-level anomaly classifiers.

[0176] The anomaly detection module identifies record anomalies and field anomalies based on a trained record-level anomaly classifier and a field-level classifier, and finally outputs the anomaly data.

[0177] The anomaly detection module's detection process is as follows: First, unlabeled record data is detected by inputting the encoding table DC into the trained record-level classifier to identify record-level anomalies; second, the numerical column features are input into the corresponding column's field-level classifier to detect numerical column field anomalies; finally, based on the field set Lc generated in the configuration loading module and the user's annotation results, combined with label propagation, the anomalies of most data in the classification column can be determined, and the frequency features of the remaining data are all within the calculated quartile limits, and are considered to belong to the normal category.

[0178] Based on the above system, this invention provides an anomaly identification method with propagation learning capability for data governance, comprising the following steps:

[0179] 1. Execute step S1 through the data and configuration loading module.

[0180] S1: Load the configuration file, read the data to be tested from the data source, and distinguish the data to be tested into numeric fields and categorical fields;

[0181] 2. Execute step S2 through the data preprocessing module.

[0182] S2: Preprocess the data to be detected to obtain the sampling range and initial label;

[0183] 3. Execute step S3 by having the model training module and user interaction module work together.

[0184] S3: Sampling data according to the sampling interval, and combining interactive labeling operations, to train field-level anomaly classifiers and record-level anomaly classifiers;

[0185] 4. Execute step S4 through the anomaly detection module.

[0186] S4: Use the trained field-level anomaly classifier and record-level anomaly classifier to perform field-level anomaly identification and record-level anomaly identification on the data to be detected, and output the anomaly data.

[0187] In step S1, loading the configuration file specifically involves: obtaining parameter settings and normal data information from the configuration file. Parameter settings include the path of the data to be detected, the number of samples, the feature generation algorithm, and the label propagation method. Normal data information refers to the normal records and normal category values ​​of the categorization fields known to the user on the data. The record set Lr and the field set Lc are initialized, and the normal data information is stored in the corresponding structure, or empty if there is no data. For example, Lr = {x1, x2, ...}, Lc = {j1: (c1, c2, ...), ...}, where x i It is a complete record, where c1 represents the category value in column j1.

[0188] In step S1, data reading involves reading the data to be tested, D, from the data source, distinguishing between the column names of numeric and categorical fields in the data, and storing the data and column names in the corresponding structures.

[0189] The preprocessing in step S2 specifically includes the following steps:

[0190] A1: Generate a feature set, including the following steps:

[0191] A1-1: For numerical fields, configure parameters according to the feature generation algorithm and generate features; use one or more of the IF, ECOD, and HBOS algorithms to generate features. The method of generating features is to detect each column of numerical fields, use the abnormal score of the algorithm as the feature of the field, generate a total of m-dimensional features, and use Min-Max standardization to process the features, where m is the number of algorithms used above.

[0192] A1-2: For categorical fields, use the frequency transformation method of homogeneity to generate a one-dimensional feature for each field;

[0193] A1-3: Store the features corresponding to each field in a set; data standardization is the process of unifying data of different dimensions to the same dimension range. Its purpose is to eliminate the influence of different dimensions between variables and make different indicators comparable. Min-Max standardization specifically involves:

[0194] The formula for linearly mapping the original data to the range [0,1] is as follows:

[0195]

[0196] Where min(X) and max(X) are the maximum and minimum values ​​of the data in this column, respectively;

[0197] The IF algorithm is an unsupervised machine learning algorithm mainly used to detect anomalies in numerical data. It finds outliers in the data by constructing a set of isolation trees and calculates the anomaly score of the sample based on the path length of the isolation trees.

[0198] The ECOD algorithm, also an unsupervised machine learning algorithm, first estimates the latent distribution of the data in a nonparametric manner by calculating the empirical cumulative distribution function (ECDF). Then, it uses these empirical distributions to estimate the tail probability of each data point. Finally, it calculates the outlier value of each data point by aggregating the estimated tail probabilities. The method for calculating the ECDF is as follows:

[0199]

[0200]

[0201] in, and Let be the left-tailed ECDF and right-tailed ECDF in the j-th dimension, respectively. Θ is an indicator function, which is 1 when its parameter is true and 0 otherwise. The outlier calculation method for each sample is as follows:

[0202]

[0203]

[0204]

[0205] Among them, O left (x), O right (x), O auto (x) represents the left-tailed outlier, right-tailed outlier, and corrected outlier, respectively, where d is the dimension of x and γ is the skewness of the data.

[0206] The final anomaly score for the sample is:

[0207] O(x) = max{O left (x),O right (x),O auto (x)}

[0208] The HBOS algorithm is as follows: Histogram detection HBOS is a statistical method. First, the column is normalized using the Min-Max method to scale it to the same scale. Second, the data range is divided into several intervals, the number of samples in each interval is counted, and the relative frequency of the samples in that interval is calculated. Assuming the column has k intervals, represented as B = {b1, b2, ..., b...} k}, then the relative frequency corresponding to the i-th interval is:

[0209]

[0210] Where n(i) represents the interval b i The number of samples in the sample, where n represents the total number of samples.

[0211] For each sample, map it to the corresponding interval. Suppose sample x falls in interval b. i In the meantime, the calculated anomaly score s(x) is:

[0212] s(x)=-log(p i )

[0213] The specific methods for frequency conversion and same-frequency differentiation are as follows:

[0214] The frequency conversion calculates the occurrence frequency of each category in the field and converts it into a frequency representation. The conversion method is as follows:

[0215]

[0216] Where c(i) is the i-th category in this field, and n is the total number;

[0217] Since categorized data often contain elements with the same frequency, making them indistinguishable, this invention proposes a frequency-differentiation method. This method distinguishes categories with the same frequency by adding random values, minimizing the impact on the results. First, all category frequencies are sorted in ascending order. Then, random values ​​are added to values ​​with the same frequency, ensuring that the frequency order remains unchanged after adding the random values. Let the sorted set be P = {p1, p2, ..., p...}. l}, then the random value r is selected in the following way:

[0218] r = random(p) i -p left ,p right -p i )

[0219] Where, p left It is p i The left side is not equal to p i The maximum value of p right It is p i The right side is not equal to p i The minimum value.

[0220] A2: Group identification and coding, including the following steps:

[0221] A2-1: Based on clustering methods, the features of numerical fields are converted into encoded representations of clustering results. First, a clustering model is trained for each numerical column, and the K-Means algorithm is used for clustering. After constructing the clustering model, each numerical value is replaced with its corresponding cluster label. Let the j-th column generate four clusters C. j ={c0,c1,c2,c3}, replace all data in this column with the cluster number {0,1,2,3} to which it belongs;

[0222] The specific process of clustering using the K-Means algorithm is as follows:

[0223] B1: Arbitrarily initialize k sample points from all sample points as cluster centroids;

[0224] B2: Calculate the distance between all points and the k centroids, and assign each data point to the cluster containing the nearest centroid;

[0225] B3: Recalculate the centroid of each cluster based on the newly divided points;

[0226] B4: Repeat steps B1 to B3 until the centroid converges;

[0227] The method for calculating the centroid is as follows:

[0228]

[0229] Where, μ i Let represent the i-th centroid, and ni be the number of contained points of cluster Si.

[0230] A2-2: Convert the categorical field features into coded representations; for any categorical field column, sort it according to the magnitude of its frequency features and convert it into a coded representation. For example, the category with the highest frequency in the column is coded as 0, the next highest as 1, and so on. After converting all columns, generate a coded data table DC.

[0231] A3: Obtain the sampling interval and initial label, including the following steps:

[0232] A3-1: Pre-scoring using Enhanced Isolated Forest:

[0233] To alleviate user interaction pressure, it is necessary to minimize the amount of data to be labeled. An unsupervised algorithm is used to pre-evaluate abnormal records, identifying relatively obvious normal and abnormal data. Data with higher uncertainty is sampled and then manually labeled to improve sampling effectiveness. The augmented isolated forest algorithm model provided in this invention can effectively identify mixed-type data anomalies. It calculates an anomaly score for each data point and sorts the scores from highest to lowest; a higher score indicates a more abnormal data point.

[0234] It is important to emphasize that, in response to the problem that the classic IF algorithm is only applicable to numerical data and cannot handle categorical and mixed data, this invention proposes an enhanced isolated forest algorithm, which can effectively detect anomalies in mixed data and has strong robustness, and is used for anomaly scoring of mixed data.

[0235] The specific computational steps of the augmented isolated forest algorithm are as follows:

[0236] C1: Calculate the information gain ratio of the categorical fields.

[0237] Information gain is used to measure the degree of influence of a field on the classification result. The information gain ratio is the information gain divided by the entropy of the dataset after partitioning. This can penalize features with a large number of values. The calculation method is as follows:

[0238]

[0239]

[0240]

[0241] in, and `<information_gain_ratio>` and `<information_gain_growth_ratio>` represent two different categorical fields, respectively. `<information_gain_ratio>` and `<information_gain_growth_ratio>` represent the calculation of the information gain ratio and information gain between the categorical fields, respectively. Represents calculated fields right Information gain ratio; H(·) and H(·|·) represent the calculation of information entropy and conditional entropy, respectively; v m Represents category field The m-th attribute value in p(v) m ) indicates in the field In the middle, the calculation equals v m The ratio of the number of attribute values ​​to the total length of the attribute;

[0242] C2: Calculate correlation

[0243] Because information gain ratio is asymmetric and cannot effectively measure the correlation between categorical fields, it is necessary to transform the information gain ratio to calculate the correlation between categorical fields, as shown below:

[0244]

[0245] in, Represents categorized fields and The correlation between them; the larger the value, the stronger the correlation.

[0246] C3: Feature Processing

[0247] Based on the correlation analysis results between the categorical fields in the data, significantly related categorical fields are concatenated to generate new categorical fields, and the original categorical fields are deleted, thus forming new mixed data. Features are extracted from the existing categorical fields, that is, the frequency transformation method of homogeneity is used to convert the categorical data into numerical data, enabling it to identify the mixed data;

[0248] C4: Sampling

[0249] A certain number of subsamples are randomly selected from the data; for each isolated tree, a root node is set and the root node is used as the current node;

[0250] C5: Attribute Splitting

[0251] At the current node, several dimensions are randomly selected as target dimensions for segmentation, and the left and right subtrees of the node are constructed based on the segmentation results. The segmentation strategy is as follows:

[0252]

[0253] in, This represents a vector composed of several target dimensions. The intercept vector is derived by taking values ​​from a uniform distribution between the maximum and minimum values ​​in each target dimension. The normal vector is generated randomly for each target dimension using a standard normal distribution.

[0254] Through calculation Obtain the splitting hyperplane. If it is less than 0, that is, it is located below the splitting hyperplane. The corresponding sample falls into the left subtree, and the rest fall into the right subtree.

[0255] C6: Constructing an Enhanced Isolated Tree

[0256] Repeat step C5 in the child nodes until there is only one piece of data in the child node or the child node has reached the maximum depth of the set tree, then stop the step and complete the construction of the isolated tree;

[0257] C7: Building Enhanced Isolated Forests

[0258] Based on steps C4 to C6 above, construct multiple reinforced isolated trees to form an reinforced isolated forest;

[0259] C8: Calculate anomaly scores

[0260] Input the dataset processed in step C3 into the augmented isolation forest. For each data point, calculate its average path length in the isolation forest using the following formula:

[0261]

[0262] Where E(h(x)) represents the average path length, T represents the number of trees in the augmented isolated forest, and h t (x) represents the path length of the data point on tree t;

[0263] By normalizing the average path length of all data, an anomaly score can be obtained for each data point. The calculation formula is as follows:

[0264]

[0265] c(n) = 2H(n-1) - 2(n-1) / n

[0266] H(k)=ln(k)+ε

[0267] Where n represents the amount of data in the input samples for building the tree, c(n) is the global average path length used for normalization, and ε is Euler's constant, ε = 0.5772156649.

[0268] A3-2: Segment the data and determine the sampling interval

[0269] Based on a set threshold (usually set to 1% to 5%), the data is split according to the score of the enhanced isolated forest. The group labels near the threshold are the most difficult to distinguish. After splitting, the abnormal data set A and the normal data set N are obtained.

[0270] Remove duplicates from sets A and N to obtain duplicate sets A' and N'. Then, further divide sets A' and N' into 50% segments, denoted as AC, AI, NI, and NC. The latter 50% of set A' and the former 50% of set N', i.e., AI and NI, are considered to be uncertain sets because this part of the data is closer to the outlier threshold. These segments are divided into sampling intervals, while samples outside the sampling intervals, i.e., AC and NC, are considered to be deterministic samples.

[0271] A3-3: Obtaining Initial Tags

[0272] Data outside the sampling interval are labeled based on the enhanced isolated forest score to obtain initial labels. The overall labeling method is as follows:

[0273]

[0274] Where, d i It is the i-th data point in the original dataset D, l i It is d i The initial tag.

[0275] Step S3 specifically includes the following steps:

[0276] D1: Data sampling and result acquisition, including the following steps:

[0277] D1-1: Sample the data within the sampling interval.

[0278] Since each column in different records may contain the same error, in the data preprocessing module, each data column is divided into several data clusters and represented by an encoding. To minimize the number of labels required, ideally, the sampled data should cover all unlabeled clusters in each column. Therefore, sampling is performed through multiple iterations. In each iteration, the sampling probability of each data point is calculated using the Softmax function.

[0279]

[0280]

[0281] Where, p i It is d i The sampling probability, p ij It is data d i The probability of in the j-th column, assuming d i The element in column j belongs to cluster c, N jc represents the number of samples that have been taken from the c-th cluster in the j-th column, and k is the total number of clusters in the j-th column. Using this probability is beneficial for obtaining data containing clusters that have been sampled less frequently. Sampling is performed according to the sampling probability until the number of samples reaches a preset value, forming a sample set S.

[0282] D1-2: Use the quartile method to filter the categorical data to be labeled.

[0283] For each categorical field, the quartile values ​​are calculated based on frequency characteristics, and categories with frequencies below the lower quartile threshold are added to the categorical field sampling set S. l The lower quartile boundary is calculated as follows:

[0284] IQR = 1.5 * (Q3 - Q1)

[0285] b = Q1 - IQR

[0286] Q1 and Q3 are the 25th percentile and 75th percentile, respectively;

[0287] D1-3: Obtaining Tagging Results Using the User Interaction Module

[0288] Based on the user interaction module, the sampling set S and the classification field sampling set S l The data is then passed to the user for annotation, and the annotation results are parsed to obtain data labels.

[0289] D2: Train a record-level classifier through interactive active learning, including the following steps:

[0290] D2-1: Training the initial classification model

[0291] Compared with traditional decision tree models, the LightGBM model selected in this invention has higher training speed and accuracy. LightGBM is selected as the classifier, and combined with the acquired data labels, including the record set Lr generated by the data and configuration loading module, the initial labels in step S2 of the data preprocessing module, and the data labels obtained by the interaction module, the corresponding data in the encoding table DC is used as input to obtain the initially trained model.

[0292] LightGBM is a gradient boosting decision tree algorithm applicable to classification and regression tasks. It employs a gradient boosting method to iteratively improve model accuracy. The LightGBM model operates as follows: Initially, the model is a simple decision tree. In each iteration, more decision trees are added to correct previous prediction errors. A histogram algorithm is used to optimize the training process, discretizing continuous feature values ​​and dividing the dataset into multiple histograms. Within each histogram, the gradient and number of samples are calculated, and the optimal split point is selected. The gradient is calculated as follows:

[0293]

[0294] in It is the loss function, where y is the true value. These are model predictions.

[0295] D2-2: Selecting samples to be labeled based on predicted probabilities

[0296] The model is used to predict the deduplication sets A' and N', and the predicted probabilities of each sample belonging to the normal and abnormal classes are obtained. The calculation formula is as follows:

[0297]

[0298] Where P(y=k|x) represents the probability that the sample belongs to class k, and the score k The score represents the average score of category k across all decision trees. k The calculation method is as follows:

[0299]

[0300]

[0301] Where pi(k) represents the probability that a sample is assigned to a leaf node in the i-th tree and that sample belongs to class k at that leaf node, and N is the number of decision trees, n i This is the total number of nodes in this leaf node, n. i (k) is the number of leaf nodes belonging to category k;

[0302] The uncertainty probability of each sample is calculated based on the predicted probability, as follows:

[0303] prop=Abs(P(y=0|x)-0.5)+Abs(P(y=1|x)-0.5)

[0304] Where P(y=0|x) and P(y=1|x) represent the probabilities of the sample being normal and abnormal, respectively, and Abs() represents the absolute value operation;

[0305] Sort all predicted samples according to their uncertainty probability, take the 10 samples with the highest uncertainty, and update the sample set S accordingly;

[0306] D2-3: Labeled Samples

[0307] The samples obtained in the previous step are passed into the user interaction module to obtain the labels and update the label set.

[0308] D2-4: Update the model

[0309] Train a new model using all the data with existing labels;

[0310] D2-5: Repeat steps D2-2 to D2-4 until the number of samples in step D2-2 reaches the preset value, and obtain the final record-level anomaly detection classifier m0.

[0311] D3: Field Tag Propagation

[0312] To maximize the use of known information and reduce the number of labels required, this invention designs a field label propagation scheme based on encoding results. For numerical columns, existing labels are propagated to other data in the same cluster according to the clustering results to obtain additional labeled field data; for categorical fields, labels are propagated to other data with the same attribute value.

[0313] Since a cluster may have multiple normal or abnormal labels with contradictory meanings, this invention designs two propagation methods—homogeneous propagation and majority propagation. Homogeneous propagation only propagates to clusters that do not contain contradictory labels, that is, clusters containing only one type of label. Assume that for the j-th column field, there are r clusters C = {c1, c2, ..., c...} r If c i If all the labeled data labels in c are 1, then i Other data labels in the data were also propagated as 1;

[0314] Most propagation will spread user tags in clusters with mixed tags, choosing the tag with the greater prevalence in the cluster for propagation. If c i If the proportion of labeled data in the data is larger, then c i Other data labels in the data are also propagated as 1; if the proportions are the same, they are not propagated.

[0315] Based on the configuration file settings, select a propagation method and propagate the labels for each column field separately to obtain the propagated field-level labels.

[0316] D4: Training field-level classifier

[0317] For each numerical field j, filter the data with existing labels and train a LightGBM classifier m using the features generated in step S2. j We obtain l field-level anomaly classifiers M = {m1, m2, ..., m}. l}, where l represents the number of columns containing numeric fields.

[0318] Based on the above scheme, in order to verify the effectiveness and practical effect of the present invention, experimental verification was carried out, as follows:

[0319] Experiment 1:

[0320] In this embodiment, a portion of the data to be detected is as follows: Figure 2 As shown, some of the configurations used in the experiment are as follows: Figure 3 As shown.

[0321] During the experiment:

[0322] The obtained feature set and encoding conversion table are as follows: Figure 4 and Figure 5 As shown, the feature set consists of numerical field features and categorical field features. The data contains a total of 4 numerical fields and 12 categorical fields. Each numerical field generates a 3-dimensional feature, and each categorical field generates a 1-dimensional feature. The categorical fields in the coding table are generated by the magnitude of the frequency feature. For example, the code for "rural low-voltage residents" is "1", and the code for "urban low-voltage residents" is "0", indicating that the "urban low-voltage residents" category has the most occurrences. The numerical field codes are converted from the feature clustering results. For example, the code for the operating capacity field with a value of "3" is "0", indicating that its clustering result belongs to the 0th cluster.

[0323] Figure 6 The results are scored and sorted in the data preprocessing module. The "score" column is the anomaly score given by the augmented isolated forest for each record. The score range is 0-1. After sorting according to the "score" column, the record with the highest anomaly probability is listed first, and the record with the highest normal probability is listed last. The sampling interval can be obtained by splitting the dataset according to the anomaly ratio.

[0324] Figure 7 It is the process of receiving and recording sampling results and user tagging by the user interaction module, such as... Figure 7As shown, the log output displays the batch sampling data sequence numbers from the model training module. A total of 40 points were selected and assigned to the user for labeling. The data with sequence number 176 was first marked and its specific information was displayed. The user needs to label each field in this record, and then label any abnormalities in the entire record. The user labels it as 1, indicating that the record is abnormal.

[0325] Figure 8 This refers to the training process of the record-level anomaly classification model in the model training module. Figure 8 This demonstrates the specific process of active learning, which includes sub-steps such as sampling data, providing feedback labels, and updating the model in each learning phase. Through multiple iterations, the detection performance of the record-level model is gradually improved.

[0326] Figure 9 This refers to the field label propagation process in step D3 of the model training module. Using a homogeneous propagation strategy, in column 14, some classes already have a certain number of labels. For example, in class 1, the number of labels with '1' is 12 and the number of labels with '0' is 4; in class 2, the number of labels with '1' is 0 and the number of labels with '0' is 83. Using homogeneous propagation, class 1 is not propagated, and class 2's label propagation is 0. After propagation, the number of labels in all columns increases from 1600 to approximately 600,000.

[0327] The final detection and recognition results are shown in Table 1:

[0328] Table 1 Partial Test Results

[0329]

[0330]

[0331] In Table 1, the “Score” column contains the initial scores obtained using the augmented isolated forest, and the label column contains the prediction results of the record-level classifier. In each data result, the first row is the data display and its corresponding label, and the second row is the detection result of each field, where 1 is abnormal and 0 is normal.

[0332] For data 1, both its wiring method and phase field are "three-phase," which violates the... Figure 3The configuration setting in the data should actually be "three-phase four-wire," therefore it was judged as abnormal. The value of the comprehensive multiplier field is "160," which is generally high in the data and was also judged as abnormal. The record-level abnormal label is 1 because the user classification field and the electricity consumption category field do not match. Their corresponding values ​​are "urban grid low-voltage non-residential" and "urban residential electricity consumption," respectively. The actual electricity consumption category value should be "commercial electricity consumption" or other non-residential household electricity consumption types. For data 2, its user classification field does not conform to the configuration and is judged as abnormal. Furthermore, the user classification field and the electricity consumption category field cannot match, so this record is also abnormal. For data 4, all its fields are normal, but there is a pattern abnormality between the wiring method and the phase line field. "Single-phase" does not correspond to "three-phase four-wire," so this record was detected as abnormal.

[0333] The initial scores generated in step A3-1 of the data preprocessing module can reflect the anomalies in the data to some extent. For example, the scores of data 1, data 2, and data 4 are all at a high level, indicating a higher probability of anomalies, and the final identification results are all anomaly data. However, the initial scoring is an unsupervised algorithm and may have some bias. By using the semi-supervised processing in the model training module to correct the initial scoring results, more accurate anomaly labels can be obtained, such as for data 3 and data 6.

[0334] Experiment 2:

[0335] Table 1 Comparison of Record-Level Detection Results

[0336]

[0337]

[0338] In this embodiment, a record-level anomaly detection classifier is trained through steps such as initial scoring, sampling labeling, and active learning. Table 1 compares the experimental results after removing certain steps. The initial scoring experiment ranked the data based on scores, with the top 1% marked as anomalies. The classification without active learning involved labeling and sampling after initial scoring, followed by direct classifier training. Table 1 shows that relying solely on initial scoring yielded the worst result, with an F1 score of only 0.62. After adding the labeling and sampling process to train the classifier, the F1 score improved by 0.1. Furthermore, after multiple model updates through active learning, the F1 score improved by 0.12, reaching 0.84. This demonstrates that the labeling method combines the advantages of both unsupervised and supervised methods, improving detection accuracy, and that active learning further corrects the results.

[0339] Table 2 Comparison of Field-Level Detection Results

[0340] Unlabeled transmission Label-based dissemination Precision 0.73 0.95 Recall 0.88 0.80 F1 0.80 0.87

[0341] Table 2 compares the performance of field-level detection classifiers. Unlabeled propagation directly trains the field-level classifier based on the sampled label results. It can be seen that unlabeled propagation has a higher recall but lower precision, indicating that it detects many false anomalies. Labeled propagation, on the other hand, has a higher F1 score and is more stable overall, suggesting that expanding the amount of labeled data through propagation makes the classifier more robust.

[0342] Based on Tables 1 and 2, it can be seen that the present invention integrates multiple types of anomaly detection classifiers into the system, and simultaneously identifies multiple anomaly types contained in mixed data, thus effectively dealing with different anomaly types.

Claims

1. An anomaly identification method with propagation learning capability for data governance, characterized in that, Includes the following steps: S1: Load the configuration file, read the data to be tested from the data source, and distinguish the data to be tested into numeric fields and categorical fields; S2: Preprocess the data to be detected to obtain the sampling range and initial label; S3: Sampling data based on the sampling interval, combined with interactive labeling operations, to train field-level anomaly classifiers and record-level anomaly classifiers; S4: Use the trained field-level anomaly classifier and record-level anomaly classifier to perform field-level anomaly identification and record-level anomaly identification on the data to be detected, and output the anomaly data. Step S3 specifically includes the following steps: D1: Data sampling and result acquisition, including the following steps: D1-1: Sample the data within the sampling interval. Sampling is performed through multiple iterations. In each iteration, the sampling probability of each data point is calculated using the Softmax function. Where, p i It is d i The sampling probability, p ij It is data d i The probability in the j-th column, assuming d i The element in column j belongs to cluster c, N jc represents the number of samples that have been collected from the c-th cluster in the j-th column, and k is the total number of clusters in the j-th column; sampling is performed according to the sampling probability until the number of samples reaches a preset value, forming a sample set S; D1-2: Use the quartile method to filter the categorical data to be labeled. For each categorical field, the quartile values ​​are calculated based on frequency characteristics, and categories with frequencies below the lower quartile threshold are added to the categorical field sampling set S. l The lower quartile boundary is calculated as follows: IQR = 1.5 * (Q3 - Q1) b = Q1 - IQR Q1 and Q3 are the 25th percentile and 75th percentile, respectively; D1-3: Obtaining the marking results using the interactive module Based on the user interaction module, the sampling set S and the classification field sampling set S l The data is then passed to the user for annotation, and the annotation results are parsed to obtain data labels. D2: Train a record-level classifier through interactive active learning, including the following steps: D2-1: Training the initial classification model LightGBM is selected as the classifier. Combined with the acquired data labels, including the record set Lr generated by the data and configuration loading module, the initial labels in step S2, and the data labels obtained by the interaction module, the corresponding data in the encoding table DC is used as input to obtain the initially trained model. D2-2: Selecting samples to be labeled based on predicted probabilities The model is used to predict the deduplication sets A' and N', and the predicted probabilities of each sample belonging to the normal and abnormal classes are obtained. The calculation formula is as follows: Where P(y=k|x) represents the probability that the sample belongs to class k, and the score k The score represents the average score of category k across all decision trees. k The calculation method is as follows: Where, p i (k) represents the probability that a sample is assigned to a leaf node in the i-th tree and that sample belongs to class k at that leaf node, where N is the number of decision trees and n i This is the total number of nodes in this leaf node, n. i (k) is the number of leaf nodes belonging to category k; The uncertainty probability of each sample is calculated based on the predicted probability, as follows: prop=Abs(P(y=0|x)-0.5)+Abs(P(y=1|x)-0.5) Where P(y=0|x) and P(y=1|x) represent the probabilities of the sample being normal and abnormal, respectively, and Abs() represents the absolute value operation; Sort all predicted samples according to their uncertainty probability, take the 10 samples with the highest uncertainty, and update the sample set S accordingly; D2-3: Labeled Samples The samples obtained in the previous step are passed into the user interaction module to obtain the labels and update the label set. D2-4: Update the model Train a new model using all the data with existing labels; D2-5: Repeat steps D2-2 to D2-4 until the number of samples in step D2-2 reaches the preset value, and obtain the final record-level anomaly detection classifier m0; D3: Field Tag Propagation Design a field label propagation scheme based on encoding results. For numeric columns, propagate existing labels to other data in the same cluster according to the clustering results to obtain additional labeled field data; for categorical fields, propagate labels to other data in the same category. D4: Training field-level classifier For each numerical field j, filter the data with existing labels and train a LightGBM classifier m using the features generated in step S2. j We obtain l field-level anomaly classifiers M = {m1, m2, ..., m}. l }, where l is the number of columns of numeric fields contained in the data.

2. The anomaly identification method with propagation learning capability for data governance according to claim 1, characterized in that, The configuration file loading in step S1 specifically involves: obtaining parameter settings and normal data information from the configuration file. Parameter settings include the data path to be detected, the number of samples, the feature generation algorithm, and the label propagation method. Normal data information refers to the normal records and normal category values ​​of the categorization fields known to the user on the data. The record set Lr and the field set Lc are initialized, and the normal data information is stored in the corresponding structure, or empty if there is no normal data. Lr = {x1, x2, ... x...} i }, Lc={j1:(c1,c2,…),…}, where x i It is a complete record, where c1 represents the category value in column j1.

3. The anomaly identification method with propagation learning capability for data governance according to claim 1, characterized in that, The preprocessing in step S2 specifically includes the following steps: A1: Generate a feature set, including the following steps: A1-1: For numerical fields, configure parameters according to the feature generation algorithm and generate features; use one or more of the following algorithms to generate features: Isolation Forest, Empirical Cumulative Distribution Detection Algorithm, and Histogram Detection Algorithm. The method for generating features is to detect each column of numerical fields, use the outlier score of the algorithm as the feature of the field, generate a total of m-dimensional features, and then use Min-Max standardization to process the features, where m is the number of times the above algorithms are used. A1-2: For categorical fields, use the frequency transformation method of homogeneity to generate a one-dimensional feature for each field; A1-3: Store the features corresponding to each field in a collection; A2: Group identification and coding, including the following steps: A2-1: Based on clustering methods, the features of numerical fields are converted into encoded representations of clustering results. First, based on the generated features, a K-Means algorithm is used to train a clustering model for each numerical column. After constructing clustering models for all columns, the values ​​in each column are replaced with the corresponding clustering labels. Let C be the generated clustering label for the j-th column. j If the expression is {c0,c1,c2,c3}, then all data in that column will be replaced with the cluster number {0,1,2,3} to which it belongs. A2-2: Convert the categorical field features into coded representations; for any categorical field column, convert it into a coded representation based on its frequency feature size, such as the category with the highest frequency in the column being coded as 0, the next highest as 1, and so on. A2-3: Concatenate the encoding results of the numeric field and the categorical field to generate the encoded data table DC; A3: Obtain the sampling interval and initial label, including the following steps: A3-1: Use the augmented isolated forest algorithm to calculate the anomaly score for each data point and sort the scores from high to low. The higher the score, the more abnormal the data point is. A3-2: Segment the data and determine the sampling interval Based on a set threshold, the data is split according to the score of the enhanced isolated forest, resulting in an abnormal data set A and a normal data set N. Remove duplicates from sets A and N to obtain duplicate sets A' and N'. Then further divide sets A' and N' into 50% sets, denoted as AC, AI, NI, and NC. Consider the last 50% of set A' and the first 50% of set N', i.e., AI and NI, as uncertain sets and divide them into sampling intervals. The samples outside the sampling intervals, i.e., AC and NC, are considered as deterministic samples. A3-3: Obtaining Initial Tags Data outside the sampling interval are labeled based on the enhanced isolated forest score to obtain initial labels. The overall labeling method is as follows: Where, d i It is the i-th data point in the original dataset D, l i It is d i The initial tag.

4. The anomaly identification method with propagation learning capability for data governance according to claim 3, characterized in that, The Min-Max standardization in step A1 specifically involves: The formula for linearly mapping the original data to the range [0,1] is as follows: Where min(X) and max(X) are the maximum and minimum values ​​of the data in this column, respectively; The Isolation Forest algorithm specifically involves: constructing a set of isolation trees to find isolated points in the data, and calculating the outlier score of the samples based on the path length of the isolation trees; The Empirical Cumulative Distribution Detection (ECDF) algorithm works as follows: First, the latent distribution of the data is estimated in a non-parametric manner by calculating the Empirical Cumulative Distribution Function (ECDF). Second, these empirical distributions are used to estimate the tail probability of each data point. Finally, the outlier value of each data point is calculated by aggregating the estimated tail probabilities. The method for calculating the Empirical Cumulative Distribution Function (ECDF) is as follows: in, and Let be the left-tailed ECDF and right-tailed ECDF in the j-th dimension, respectively. Θ is an indicator function, which is 1 when its parameter is true and 0 otherwise. The outlier calculation method for each sample is as follows: Among them, O left (x), O right (x), O auto (x) represents the left-tailed outlier, right-tailed outlier, and corrected outlier, respectively, where d is the dimension of x and γ is the skewness of the data. The final anomaly score for the sample is: O(x)=max{O left (x),O right (x),O auto (x)} The histogram detection algorithm works as follows: First, the column is normalized using the Min-Max method to scale it to the same scale. Second, the data range is divided into several intervals, the number of samples in each interval is counted, and the relative frequency of the samples in that interval is calculated. Assuming the column has k intervals, represented as B = {b1, b2, ..., b...} k }, then the relative frequency corresponding to the i-th interval is: Where n(i) represents the interval b i The number of samples in the sample, where n represents the total number of samples; For each sample, map it to the corresponding interval. Suppose sample x falls in interval b. i In the meantime, the calculated anomaly score s(x) is: s(x)=-log(p i )。 5. The anomaly identification method with propagation learning capability for data governance according to claim 3, characterized in that, The frequency conversion and same-frequency differentiation method in step A1 is as follows: The frequency conversion calculates the occurrence frequency of each category in the field and converts it into a frequency representation. The conversion method is as follows: Where c(i) is the i-th category in this field, and n is the total number; A frequency-differentiation method is proposed, which distinguishes categories with the same frequency by adding random values ​​while minimizing the impact on the results. First, the frequencies of all categories are sorted in ascending order. Then, random values ​​are added to values ​​with the same frequency, ensuring that the frequency order is not changed after adding the random values. Let the sorted set be P = {p1, p2, ..., p...} l }, then the random value r is selected in the following way: r=random(p i -p left ,p right -p i ) Where, p left It is p i The left side is not equal to p i The maximum value of p right It is p i The right side is not equal to p i The minimum value.

6. The anomaly identification method with propagation learning capability for data governance according to claim 3, characterized in that, The specific process of clustering using the K-Means algorithm in step A2-1 is as follows: B1: Arbitrarily initialize k sample points from all sample points as cluster centroids; B2: Calculate the distance between all points and the k centroids, and assign each data point to the cluster containing the nearest centroid; B3: Recalculate the centroid of each cluster based on the newly divided points; B4: Repeat steps B1 to B3 until the centroid converges; The method for calculating the centroid is as follows: Where, μ i Represents the i-th centroid, n i For cluster S i The number of points contained in it.

7. The anomaly identification method with propagation learning capability for data governance according to claim 3, characterized in that, The specific calculation steps of the enhanced isolated forest algorithm in step A3-1 are as follows: C1: Calculate the information gain ratio of the categorical field, as follows: in, and These represent two different categorical fields. GainRatio(·) and Gain(·) respectively calculate the information gain ratio and information gain between the categorical fields. Represents calculated fields right Information gain ratio; H(·) and H(·|·) represent the calculation of information entropy and conditional entropy, respectively; v m Represents category field The m-th attribute value in p(v) m ) indicates in the field In the middle, the calculation equals v m The ratio of the number of attribute values ​​to the total length of the field; C2: Calculate the correlation between categorical fields, as shown below: in, Represents categorized fields and The correlation between them; C3: Feature Processing Based on the correlation analysis results between the categorical fields in the data, the significantly related categorical fields are concatenated together to generate new categorical fields, and the original categorical fields are deleted to form new mixed data. Features are extracted from the existing categorical fields, that is, the categorical data is converted into numerical data using the frequency transformation method of homogeneity, so that it can identify the mixed data. C4: Sampling Randomly select subsamples from the data; for each isolated tree, set a root node and use the root node as the current node; C5: Attribute Splitting At the current node, several dimensions are randomly selected as target dimensions for segmentation, and the left and right subtrees of the node are constructed based on the segmentation results. The segmentation strategy is as follows: in, This represents a vector composed of several target dimensions. The intercept vector is derived by taking values ​​from a uniform distribution between the maximum and minimum values ​​in each target dimension. The normal vector is generated randomly for each target dimension using a standard normal distribution. Through calculation Obtain the splitting hyperplane. If it is less than 0, that is, it is located below the splitting hyperplane. The corresponding sample falls into the left subtree, and the rest fall into the right subtree. C6: Constructing an Enhanced Isolated Tree Repeat step C5 in the child nodes until there is only one piece of data in the child node or the child node has reached the maximum depth of the set tree, then stop the step and complete the construction of the isolated tree; C7: Building Enhanced Isolated Forests Based on steps C4 to C6 above, construct multiple reinforced isolated trees to form an reinforced isolated forest; C8: Calculate anomaly scores Input the dataset processed in step C3 into the augmented isolation forest. For each data point, calculate its average path length in the isolation forest using the following formula: Where E(h(x)) represents the average path length, T represents the number of trees in the augmented isolated forest, and h t (x) represents the path length of the data point on tree t; By normalizing the average path length of all data, an anomaly score can be obtained for each data point. The calculation formula is as follows: c(n) = 2H(n-1) - 2(n-1) / n H(k)=ln(k)+ε Where n represents the amount of data in the input samples for building the tree, c(n) is the global average path length used for normalization, and ε is Euler's constant.

8. The anomaly identification method with propagation learning capability for data governance according to claim 1, characterized in that, The LightGBM model operation process in step D2-1 is as follows: Initially, the model is a simple decision tree. Then, in each iteration, more decision trees are gradually added to correct the prediction errors of the previous model. A histogram algorithm is used to optimize the training process of the decision trees, discretizing the values ​​of continuous features and dividing the dataset into multiple histograms. In each histogram, the gradient and number of samples within that histogram are calculated, and then the optimal split point is selected. The gradient is calculated as follows: in It is the loss function, where y is the true value. These are model predictions.

9. An anomaly detection system with propagation and learning capabilities for data governance, characterized in that, To implement the method of claim 1, the system includes a data and configuration loading module, an algorithm library integration module, a data preprocessing module, a user interaction module, a model training module, and an anomaly recognition module; The data and configuration loading module is used to perform the loading of configuration files and loading of data in step S1; The algorithm library integration module is used to store various algorithms required by other modules for easy access at any time; The data preprocessing module is used to perform step S2; The user interaction module is used to receive sampling result data from other modules of the system, display the sampling results to the user and perform annotation operations, and finally receive the user's annotation results and save them in the corresponding structure. The model training module and the user interaction module work together to execute step S3, obtain sampled data and labeling results, and train field-level anomaly classifier and record-level anomaly classifier. The anomaly identification module is used to execute step S4, which identifies record anomalies and field anomalies based on the trained record-level anomaly classifier and field-level classifier, and finally outputs the anomaly data.

Images