Concept drift detection and interpretation method based on KL divergence and permutation test

Abstract

The invention discloses an anomaly detection and drift interpretation method based on KL divergence and permutation test. The method comprises the following steps: training an auto-encoder by using a zero positive sample set; compressing a to-be-detected sample into a low-dimensional representation by using an auto-encoder, and performing anomaly detection by using an isolated forest; after a system key event occurs, calculating the KL divergence of an event sample and a zero positive sample set, and performing drift detection by using a replacement test method; estimating the local density of the sample through a kernel density estimation method, and determining a drift sample; the importance of the features is calculated through an isolated forest algorithm, so that the cause of drift is explained; according to the method, the problems of low drift detection efficiency and lack of drift explanation in the prior art are solved, the drift detection efficiency is improved, drift reason explanation is provided, and the method can be used for real-time anomaly detection and data change monitoring scenes of various devices.

Description

Technical Field

[0001] The present invention belongs to the technical field of electronic data processing, and further relates to a concept drift detection and interpretation method based on Kullback-Leibler divergence and permutation test in the technical field of electronic data detection. The present invention can be used to detect concept drift generated in electronic data processing and interpret the detection results. Background Art

[0002] In the field of anomaly detection, concept drift refers to the phenomenon that the statistical characteristics or relationships of data change over time. This change may be caused by changes in the external environment, the evolution of the data generation process, observation bias or noise. The impact of concept drift on the model is mainly reflected in the change of data distribution caused by it, which may cause the model to fail to effectively capture new abnormal patterns or changes when facing new data distribution. This may lead to model degradation, decision bias and difficulty in model updating. At present, concept drift detection and explanation are mainly based on the error rate perspective. This type of method detects drift by monitoring the change of classifier error rate. For example, the drift detection algorithm DDM (Drift Detection Method) algorithm determines whether drift occurs by comparing the current error rate with the preset threshold. The advantage of this method is that it is simple to implement, but the disadvantage is that it is not sensitive to sudden drift changes and is prone to misjudgment when the data itself fluctuates greatly. In addition, there are also detection based on the perspective of data distribution. This type of method directly compares the changes in data distribution, such as using KL divergence or chi-square test. They can capture subtle changes in data distribution, but the computational complexity is high and there are certain assumptions about the distribution form of data.

[0003] Shandong Provincial Computing Center disclosed a concept drift detection method for industrial control systems in its patent application "An adaptable and interpretable industrial control system anomaly detection method for concept drift" (application number 202310809566.8 and application publication number CN 116991137 A). The method obtains industrial control data samples from different periods, including historical data and new data, for training anomaly detection models and saving training parameters. The output results of the anomaly detection model are calibrated, and drift detection is performed to determine whether there is concept drift in the data stream. The detected drift is explained to facilitate understanding of the reasons for the data changes. The new samples with concept drift are combined with the still relevant parts of the old samples, and the anomaly detection model is retrained to adapt to changes in data distribution. The advantages of this patented method are that it can judge and adapt to concept drift, reduce the false alarm rate of the model, and improve the interpretability of the model by explaining the reasons for the drift. However, the method still has some shortcomings. It requires large computing resources to process and analyze a large amount of industrial control data during the detection process, and in practical applications, it is necessary to continuously adjust and optimize the model parameters to adapt to the ever-changing data environment. In addition, for rapidly changing industrial control systems, the real-time performance and accuracy of this method face challenges.

[0004] Zhejiang University Jiaxing Research Institute disclosed a concept drift detection method in its patent application document "A method and device for mitigating concept drift based on unsupervised learning" (application number 202311825338.6 application publication number CN 117807437 A). The implementation steps of this method are: 1) train an unsupervised learning autoencoder through the training samples of the malicious application classifier, and use the autoencoder to calculate the distance between the sample to be tested and the center of each training sample category to realize drift sample detection; 2) for the detected drift samples, use feature attribution technology to explain the cause of sample drift based on the contribution of each feature to drift detection; 3) select drift samples and explanation results for annotation, and use the active learning framework to update the malicious application classifier; 4) use the updated malicious application classifier to classify the software. This method experimentally tests the real-time detection of sample drift by fitting the autoencoder of the training samples, and adds an interpretable module to calculate the feature contribution using the embedded distance, which greatly reduces the manual annotation cost in the active learning framework. However, the method still has the disadvantage that drift detection by calculating the distance between the sample to be tested and the center of the training sample category is difficult to capture complex or nonlinear data distribution changes, especially in high-dimensional and noisy data environments, where false positives or negatives are more likely to occur. In addition, the computational complexity of the feature attribution module is high, which will affect the efficiency of large-scale real-time detection, especially when facing large-scale data, the system's resource consumption and response speed will become the biggest bottleneck.

[0005] In summary, the existing technologies have the following four obvious shortcomings in data drift detection and anomaly detection: 1. Traditional drift detection methods, such as detection based on classifier error rate, are not very sensitive, especially when the data fluctuates greatly, they are prone to false alarms, and cannot effectively capture small distribution changes, affecting detection accuracy. 2. Existing technologies can often only detect the occurrence of data drift, but cannot clearly identify the specific causes or samples and features that cause the drift, and lack the ability to explain the causes of the drift. 3. When processing large-scale, high-dimensional data, the existing detection methods have high computational complexity and high resource consumption, resulting in low detection efficiency and inability to achieve real-time response. 4. Most existing technologies rely on a large amount of labeled data, perform poorly in an unsupervised learning environment, and cannot flexibly adapt to drift detection of unlabeled data. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies of the above-mentioned existing technologies and propose a concept drift detection and interpretation method based on KL divergence and permutation test to solve the problems of low detection accuracy, lack of drift interpretation ability, low computational efficiency and inability to flexibly adapt to drift detection of unlabeled data in the existing technologies.

[0007] The technical idea for achieving the purpose of the present invention is: the present invention combines the nonlinear feature extraction capability of the autoencoder with the efficient anomaly detection of the isolation forest, and can more keenly capture the distribution of small changes in the detected data. The dimensionality reduction capability of the autoencoder can be used to process complex data of high dimensions. This combination improves the accuracy of detection, especially in the case of large data fluctuations, and can effectively reduce false positives. In this way, the problem of low detection accuracy and difficulty in capturing subtle data changes in traditional drift detection methods is solved. The present invention adopts a new concept drift detection mechanism, which first preliminarily judges the possibility of drift by quickly calculating the KL divergence, and then accurately detects drift by permutation test. This two-step detection mechanism not only improves the accuracy of detection, but also makes judgments in a relatively short time. In this way, the problems of high computational complexity and poor real-time response capability of traditional methods are solved, and the detection efficiency is improved. The present invention establishes a feature importance analysis mechanism by evaluating the feature contribution of each decision tree in the isolation forest. This mechanism can not only explain the cause of drift, but also clearly point out which features are most critical to anomaly detection, and provide a clear drift interpretation capability. In this way, the problem of lack of drift interpretation capability and inability to identify the specific causes or key features of drift in the prior art is solved. The present invention uses the kernel density estimation method to screen out abnormal samples with excessively high local density, thereby identifying samples that are likely to be drift samples. This process does not rely on labeled data, but directly uses the distribution and density characteristics of the detected data to detect concept drift. In addition, the isolation forest evaluates the feature contribution of each tree and automatically identifies important features in the drift data, so that the model can effectively adapt to data drift in an unlabeled environment. This solves the problem that the prior art performs poorly in an unsupervised learning environment and relies heavily on a large amount of labeled data, ensuring that data drift can still be accurately detected and its causes explained in the absence of labeled data, thereby reducing the need and reliance on manual labeling.

[0008] The present invention comprises the following steps:

[0009] Step 1, train the autoencoder using the zero-positive sample set;

[0010] Step 2: Use the trained autoencoder to extract low-dimensional features of the data to be detected, and use the isolation forest method to determine whether the data to be detected is abnormal data;

[0011] Step 3, calculate the KL divergence between the distribution of the data to be tested and the distribution of the training set data and perform a permutation test to determine whether the data to be tested has drifted;

[0012] Step 4, using the kernel density estimation method, identify all drift samples in each high-density area;

[0013] In step 5, the feature contribution of each decision tree in the isolation forest is evaluated to account for the drift samples.

[0014] Compared with the prior art, the present invention has the following advantages:

[0015] 1. The present invention adopts the nonlinear feature extraction capability of the autoencoder combined with the isolation forest to form an efficient anomaly detection method, which can more keenly capture the slight distribution changes in the data, especially in the case of large data fluctuations, effectively reducing false positives. Compared with the traditional detection method based on the error rate of the classifier, the present invention can overcome the problem of low sensitivity of the traditional method through efficient feature extraction and anomaly detection, so that the present invention has the advantage of more accurately detecting concept drift and anomalies.

[0016] Second, the present invention adopts a dual detection mechanism combining KL divergence and permutation test. By using KL divergence to quickly screen out areas where drift may occur, and then performing accurate permutation tests, not only the detection accuracy is improved, but also the computational complexity is significantly reduced. Compared with the prior art, the defects of high computational complexity and high resource consumption of traditional methods when processing large-scale, high-dimensional data are overcome, so that the present invention can efficiently detect data drift, significantly improve the real-time response capability, and expand the scope of application of detecting concept drift and abnormal point data.

[0017] 3. The present invention provides a feature importance analysis mechanism by evaluating the feature contribution of each decision tree in the isolation forest. This mechanism can identify the most critical features for anomaly detection and explain the causes of drift. Through this analysis mechanism, the defect of the prior art that the cause of drift cannot be explained is overcome, so that the present invention can clearly point out which features and samples have the greatest impact on drift, thereby enhancing the interpretability of the model and helping users understand potential abnormal patterns in the data.

[0018] 4. The present invention uses an unsupervised learning model, combined with the powerful feature extraction ability of the autoencoder and the anomaly detection ability of the isolation forest, to effectively detect concept drift without relying on a large amount of labeled data. By using kernel density estimation to identify abnormal samples in high-density areas, manual labeling is avoided, reducing the cost of manual labeling, overcoming the defects of the prior art that it performs poorly in an unsupervised environment and relies on a large amount of labeled data, so that the present invention can still efficiently identify drift samples without labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of the implementation of the present invention;

[0020] Figure 2 It is a flow chart of the implementation of drift detection and interpretation of the present invention. DETAILED DESCRIPTION

[0021] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0022] Reference Figure 1 , the implementation steps of the embodiment of the present invention are further described in detail.

[0023] Step 1: Train the autoencoder using the zero-positive sample set.

[0024] The zero-positive sample set in the embodiment of the present invention is a data set containing zero-positive normal samples, with a size of 10,000 and 100 feature dimensions.

[0025] The autoencoder is L 0 ,L 1 ,L 2 ,L 3 ,L 4 ,L 5 ,L 6 ,L 7 The neural network model consists of 8 layers in series, where L 0 ,L 1 ,L 2 ,L 3 The L layer is used to encode and compress data and extract data features. The neuron size of the encoding layer decreases step by step, and the number of each decrease is one-fourth of the total number of extracted features. 4 ,L 5 ,L 6 L 7 The layer is used to decode and restore the extracted data features. Each layer contains an activation layer for linear rectification to obtain reconstructed data. The neuron size of the decoding layer increases step by step, and the number of each increase is one-fourth of the total number of features. The ReLU function is selected as the activation function.

[0026] The training autoencoder refers to: inputting a zero-positive sample set into an unsupervised learning autoencoder, the intermediate layer of the neural network model outputs an intermediate embedding of a low-dimensional space, and the output layer outputs the reconstructed training sample features; using the back propagation method, iteratively updating the parameters of the neural network model until the mean square error loss MSE of the training sample and the reconstructed training sample converges, thereby obtaining a trained autoencoder.

[0027] The mean square error loss MSE is obtained by the following formula:

[0028]

[0029] Among them, n represents the total number of samples in the zero-positive sample set, y i represents the true value of the i-th zero-positive sample, Represents the predicted value of the i-th zero-positive sample output by the neural network model.

[0030] Step 2: Use the trained autoencoder to extract low-dimensional features of the data to be detected, and use the isolation forest method to determine whether the data to be detected is abnormal data.

[0031] The steps of using the isolation forest method to determine whether the data to be detected is abnormal data are as follows:

[0032] In the first step, the data to be tested is input into the trained autoencoder to obtain a low-dimensional feature representation of the data to be tested, with a total of 25 feature dimensions. This low-dimensional feature representation retains the main structure and pattern of the input data and has a smaller feature dimension.

[0033] The second step is to generate a decision tree with a low-dimensional representation of the data to be tested. The nodes in the decision tree are represented by a feature f j and the split point θ j Composition, among which, f j represents the jth feature randomly selected from the low-dimensional representation of the data to be detected, θ j represents the split point randomly sampled from the range of the jth eigenvalue;

[0034] The third step is to recursively split the decision tree by recursively splitting the data set until the data points are completely isolated;

[0035] The fourth step is to calculate the score s(x i ,m), the low-dimensional feature data whose isolation forest score value is greater than the threshold score is judged as abnormal data; the threshold score is the score corresponding to the 1% samples with the highest isolation forest score in the zero-positive sample set.

[0036] The fraction s(x) of data points in the isolation forest i ,m) is calculated as follows:

[0037]

[0038] Among them, m represents the total number of samples of the data to be tested, h(x i ) represents the i-th sample x to be detected i The average path length of the number of steps required to be isolated in the decision tree, c(m) represents the expected path length of m, and H(m-1) is defined as: ln(·) represents the logarithmic operation with the natural constant e as the base.

[0039] Reference Figure 2 , the drift detection and interpretation process of the present invention is further described in detail.

[0040] Step 3: Calculate the KL divergence between the distribution of the data to be tested and the distribution of the training set data and perform a permutation test to determine whether the data to be tested has drifted.

[0041] The steps of calculating the KL divergence between the distribution of the data to be tested and the distribution of the zero-positive sample set data and performing a permutation test are as follows:

[0042] In the first step, the sum of the KL divergence mean and twice the standard deviation of the zero-positive sample set is set as the drift threshold, and the D difference between the data distribution of the event sample set and the data distribution of the zero-positive sample set after each system critical event is calculated. KL .

[0043] The second step is when D KL A value greater than the drift threshold indicates the presence of drift. The data ranges of the event sample set and the zero-positive sample set are binned to facilitate the analysis of data distribution.

[0044] The third step is to combine the event sample set and the zero-positive sample set into a replacement sample set. The number of replacements is set to 5000. In each replacement, the samples of the replacement sample set are randomly redistributed to the event sample set and the zero-positive sample set, and the entropy difference between the two sets is calculated.

[0045] The fourth step is to calculate the probability of a random assignment with an entropy difference greater than or equal to the initial entropy difference. The number of times the entropy difference after permutation is greater than or equal to the initial entropy difference is divided by 5000. If the result is less than 0.05, it is considered that data drift has occurred in the data to be tested.

[0046] The D KL It is obtained by the following formula:

[0047]

[0048] Where P(α) represents the frequency of occurrence of event samples in the αth interval, Q(α) represents the frequency of occurrence of zero-positive sample set in the αth interval, and ln(·) represents the logarithmic operation with the natural constant e as the base.

[0049] In step 4, all drift samples in each high-density region are identified using a kernel density estimation method.

[0050] Step 4.1: Use the Gaussian kernel density estimation method to calculate the local density of the low-dimensional representation of each event sample.

[0051] The local density It is obtained by the following formula:

[0052]

[0053] in, Represents the i-th event sample z i The density estimate of m represents the total number of samples of the data to be tested, z i represents the i-th event sample, z j represents the jth event sample, h represents the smoothness of the control density estimation, and its value is determined by cross-validation. K(·) represents the kernel function, and its expression is:

[0054]

[0055] Among them, π represents pi, e (·) Represents an exponential operation with the natural constant e as the base.

[0056] In step 4.2, the sum of the mean and twice the standard deviation of the density estimate of the zero-positive sample set is set as the density threshold, and the abnormal samples whose density estimate is greater than the density threshold are marked as drift samples.

[0057] In step 5, the feature contribution of each decision tree in the isolation forest is evaluated to account for the drift samples.

[0058] Step 5.1, calculate the average importance of the features of each decision tree in the isolation forest according to the following formula:

[0059]

[0060] Where a represents the serial number of the decision tree node in the isolation forest, M represents the total number of decision trees in the isolation forest, 1{x k is used in node n} is an indicator function. If the feature x k If it is used in the node, its value is 1, otherwise it is 0. ΔI(n) represents the change of purity at the node;

[0061] Step 5.2, find out The feature corresponding to the largest value is interpreted as the important reason why the normal sample becomes a drift sample.

Claims

1. A concept drift detection and interpretation method based on KL divergence and permutation test, characterized in that: The isolation forest method is used to determine whether the data to be detected is abnormal data, the KL divergence between the distribution of the data to be detected and the data distribution of the zero-positive sample set is calculated and a permutation test is performed to determine whether the data to be detected has drifted, and the kernel density estimation method is used to identify drift samples in high-density areas; the steps of the detection and interpretation method include the following: Step 1, train the autoencoder using the zero-positive sample set; Step 2: Use the trained autoencoder to extract low-dimensional features of the data to be detected, and use the isolation forest method to determine whether the data to be detected is abnormal data; Step 3, calculate the KL divergence between the distribution of the data to be tested and the distribution of the training set data and perform a permutation test to determine whether the data to be tested has drifted; Step 4, using the kernel density estimation method, identify all drift samples in each high-density area; In step 5, the feature contribution of each decision tree in the isolation forest is evaluated to account for the drift samples.

2. The concept drift detection and interpretation method based on KL divergence and permutation test according to claim 1, characterized in that: The autoencoder described in step 1 is a neural network model composed of 8 layers connected in series, L0, L1, L2, L3, L4, L5, L6, and L7, wherein the L0, L1, L2, and L3 layers are used to encode and compress data and extract data features. The size of neurons in the encoding layer decreases step by step, and the number of each decrease is one-fourth of the total number of extracted features; the L4, L5, L6 and L7 layers are used to decode and restore the extracted data features. Each layer includes an activation layer for linear rectification to obtain reconstructed data. The size of neurons in the decoding layer increases step by step, and the number of each increase is one-fourth of the total number of features.

3. The concept drift detection and interpretation method based on KL divergence and permutation test according to claim 1, characterized in that: The training of the autoencoder in step 1 refers to: inputting the zero-positive sample set into the autoencoder for unsupervised learning, the intermediate layer of the neural network model outputs the intermediate embedding of the low-dimensional space, and the output layer outputs the reconstructed training sample features; Using the back propagation method, the parameters of the neural network model are iteratively updated until the mean square error loss (MSE) of the training sample and the reconstructed training sample converges to obtain a trained autoencoder. The mean square error loss MSE is obtained by the following formula: Among them, n represents the total number of samples in the zero-positive sample set, y i represents the true value of the i-th zero-positive sample, Represents the predicted value of the i-th zero-positive sample output by the neural network model.

4. The concept drift detection and interpretation method based on KL divergence and permutation test according to claim 1, characterized in that: The steps of using the isolation forest method in step 2 to determine whether the data to be detected is abnormal data are as follows: In the first step, the data to be detected is input into the trained autoencoder to obtain a low-dimensional feature representation of the data to be detected. The low-dimensional feature representation retains the main structure and pattern of the input data and has a smaller feature dimension. The second step is to generate a decision tree with a low-dimensional representation of the data to be tested. The nodes in the decision tree are represented by a feature f j and the split point θ j Composition, among which, f j represents the jth feature randomly selected from the low-dimensional representation of the data to be detected, θ j represents the split point randomly sampled from the range of the jth eigenvalue; The third step is to recursively split the decision tree by recursively splitting the data set until the data points are completely isolated; The fourth step is to calculate the score s(x i ,m), the low-dimensional feature data whose isolation forest score value is greater than the threshold score is judged as abnormal data; the threshold score is the score corresponding to the 1% samples with the highest isolation forest score in the zero-positive sample set.

5. The concept drift detection and interpretation method based on KL divergence and permutation test according to claim 5, characterized in that: The fraction s(x) of data points in the isolation forest i ,m) is calculated as follows: Among them, m represents the total number of samples of the data to be tested, h(x i ) represents the i-th sample x to be detected i The average path length of the number of steps required to be isolated in the decision tree, c(m) represents the expected path length of m, and H(m-1) is defined as: ln(·) represents the logarithmic operation with the natural constant e as the base.

6. The concept drift detection and interpretation method based on KL divergence and permutation test according to claim 1, characterized in that: The steps of calculating the KL divergence between the distribution of the data to be tested and the distribution of the zero-positive sample set data and performing a permutation test in step 3 are as follows: In the first step, the sum of the KL divergence mean and twice the standard deviation of the zero-positive sample set is set as the drift threshold, and the D difference between the data distribution of the event sample set and the data distribution of the zero-positive sample set after each system critical event is calculated. KL ; The second step is when D KL A value greater than the drift threshold indicates the presence of drift, and the data range of the event sample set and the zero-positive sample set are binned to facilitate the analysis of data distribution; The third step is to combine the event sample set and the zero-positive sample set into a replacement sample set. The number of replacements is set to 5000. In each replacement, the samples of the replacement sample set are randomly redistributed to the event sample set and the zero-positive sample set, and the entropy difference between the two sets is calculated. The fourth step is to calculate the probability of a random assignment with an entropy difference greater than or equal to the initial entropy difference. The number of times the entropy difference after permutation is greater than or equal to the initial entropy difference is divided by 5000. If the result is less than 0.05, it is considered that data drift has occurred in the data to be tested.

7. The method for detecting and explaining concept drift based on KL divergence and permutation test according to claim 7, characterized in that: The D KL It is obtained by the following formula: Where P(α) represents the frequency of occurrence of event samples in the αth interval, Q(α) represents the frequency of occurrence of zero-positive sample set in the αth interval, and ln(·) represents the logarithmic operation with the natural constant e as the base.

8. The concept drift detection and interpretation method based on KL divergence and permutation test according to claim 1, characterized in that: The steps described in step 4 to identify drifting samples in high-density areas by using kernel density estimation are as follows: In the first step, the Gaussian kernel density estimation method is used to calculate the local density of the low-dimensional representation of each event sample. In the second step, the sum of the mean and twice the standard deviation of the density estimate of the zero-positive sample set is set as the density threshold, and the abnormal samples with density estimates greater than the density threshold are marked as drift samples.

9. The concept drift detection and explanation method based on KL divergence and permutation test according to claim 8, characterized in that: The local density It is obtained by the following formula: in, Represents the i-th event sample z i The density estimate of m represents the total number of samples of the data to be tested, z j represents the jth event sample, h represents the smoothness of the control density estimation, and its value is determined by cross-validation. K(·) represents the kernel function, and its expression is: Among them, π represents pi, e (·) Represents an exponential operation with the natural constant e as the base.

10. The concept drift detection and explanation method based on KL divergence and permutation test according to claim 1, characterized in that: The steps to explain the drift samples by evaluating the feature contribution of each decision tree in the isolation forest described in step 5 are as follows: The first step is to calculate the average importance of the features of each decision tree in the isolation forest according to the following formula: Where a represents the serial number of the decision tree node in the isolation forest, M represents the total number of decision trees in the isolation forest, 1{x k isused in node n} is an indicator function. If the feature x k If it is used in the node, its value is 1, otherwise it is 0. ΔI(n) represents the change of purity at the node; The second step is to find The feature corresponding to the largest value is interpreted as the important reason why the normal sample becomes a drift sample.

Images