Variational auto-encoder anomaly detection algorithm based on dynamic sparse attention mechanism

Abstract

The invention provides a variational auto-encoder anomaly detection algorithm NK-DSAVE based on a dynamic sparse attention mechanism, and the algorithm optimizes the calculation efficiency of the model while keeping high precision. Firstly, an enhanced few-sample data preprocessing method based on kernel density guidance is put forward to solve the problem of normal data category imbalance; secondly, providing a lightweight network DSCAVE, and balancing the precision and speed of the model by using an auto-encoder of a dynamic sparse attention mechanism and an auto-encoder of a dynamic skipping mechanism; and finally, providing a kernel density estimation-based dynamic selection threshold and sparse attention mechanism weighted scoring algorithm, and solving the problem of increased misjudgment rate caused by data fluctuation of small and medium-sized factory production data, thereby improving the accuracy and reliability of anomaly detection.

Description

Technical Field

[0001] The present invention relates to the field of industrial control safety, and in particular to a variational autoencoder anomaly detection algorithm based on a dynamic sparse attention mechanism. Background Art

[0002] Manufacturing is the main support of the national economy and the main battlefield for my country's economic "innovation-driven, transformation and upgrading" in the future. As a product of the integration of traditional industry and Internet technology, the industrial Internet provides a foundation for industrial digitalization and intelligence. Despite this, my country's small and medium-sized manufacturing enterprises face many challenges in this process, including system defects and insufficient intelligence, which affect their sustainable development capabilities. With the integration and development of industrial Internet and automatic control technology, the pace of the digital age has become more determined. Against this background, more solutions for smart factory upgrades have gradually appeared in the market. However, most of these intelligent systems are designed to meet the needs of large factories. For small and medium-sized factories, due to the limitations of funds and computing power, it is difficult to successfully apply the anomaly detection solutions in these intelligent systems. Therefore, the development of lightweight anomaly detection algorithms that can both ensure detection accuracy and have low system consumption is of great practical significance for promoting the intelligent upgrade of small and medium-sized enterprise factories.

[0003] At present, lightweight anomaly detection algorithms for multivariate time series data can be mainly summarized into three categories: the first category is based on decomposition and aggregation methods, which decomposes multivariate data and regards it as an independent univariate series for anomaly detection, and then aggregates the anomalies according to the corresponding outlier aggregation algorithm to determine whether the multivariate time series data is abnormal. For example, Hundman K et al. proposed to use long short-term memory deep learning technology to find point anomalies within each variable in a multivariate time series. This detection method fails to consider the interaction between each time series sensor, and only combines the anomaly detection results of a single variable without considering the possible correlation and interaction between variables, which may lead to false detection or missed detection. The second category is based on prediction methods, which use historical data to predict future data points. When the difference between the predicted value and the actual observed value exceeds a certain threshold, the data point is considered to be abnormal. Munir et al. used a CNN prediction model to detect point anomalies in multivariate time series. This method uses a part of the time series data to train the CNN model to predict future data points. Then, in the anomaly detection stage, the system compares the predicted value with the actual observed value to find the anomaly point. However, this type of method is insufficient in effectively capturing the complex characteristics of the time series. The third category is based on reconstruction models. Anomaly detection methods based on autoencoders and generative adversarial networks reconstruct input data by learning the distribution of normal data to detect anomalies. The advantage of this type of method is that it can process high-dimensional data and capture the deep features of the data, and is suitable for complex and high-dimensional data scenarios. However, reconstruction-based methods may have unstable training, and if the model structure is simplified, the accuracy will decrease.

[0004] Through the above research, we can find that the existing lightweight anomaly detection models have certain shortcomings. The first type of method fails to effectively capture and utilize the interdependencies and interactions between variables, which leads to poor performance of the model when dealing with multivariate time series data with complex relationships. The second type of prediction model requires frequent updates, and the model structure becomes more complex when dealing with more complex data sets. The third type of simple reconstruction model has problems with insufficient accuracy and unstable training. These problems highlight the need to find a balance between improving accuracy and reducing resource consumption. Summary of the invention

[0005] The purpose of this invention is to propose an anomaly detection algorithm NK-DSAVE based on a variational autoencoder with dynamic sparse attention mechanism, which ensures high detection accuracy while optimizing the computational efficiency of the model.

[0006] The object of the present invention is achieved in this way:

[0007] Firstly, a kernel density-guided enhanced few-sample data preprocessing method is used to solve the problem of imbalanced normal data categories. Secondly, a lightweight network DSAVE is constructed, and the autoencoder with dynamic sparse attention mechanism and dynamic skip mechanism is used to balance the accuracy and speed of the model. Finally, kernel density estimation is used to dynamically select thresholds and sparse attention mechanism weighted scoring algorithms to solve the problem of increased misjudgment rate caused by data fluctuations in the production data of small and medium-sized factories, thereby improving the accuracy and reliability of anomaly detection.

[0008] The specific method is as follows:

[0009] An anomaly detection algorithm based on a variational autoencoder with a dynamic sparse attention mechanism includes the following steps:

[0010] Step 1: Split the original data into windows, and then apply the kernel density estimation method to the data in the window to calculate the density score for each data point. This step aims to identify minority samples in normal data through density scores, and then increase the amount of minority sample data by adding Gaussian noise to these minority sample data.

[0011] Step 2: Normalize the sample data to ensure that all subsequent operations are performed on a unified data scale. Then use the sliding window technology to segment the normalized data;

[0012] Step 3: Data reconstruction phase. Aiming at the problem of limited computing resources in small and medium-sized factories, an improved dynamic sparse attention mechanism is proposed to reduce the model's demand for computing resources, ensuring that the model can focus on important data features, thereby improving detection accuracy;

[0013] Step 4: A dynamic skipping mechanism is proposed in the variational autoencoder, which accelerates the convergence speed by dynamically transferring the potential feature information to the decoder, and also improves the efficiency of the model in utilizing the potential features;

[0014] Step 5: Introduce a dynamic threshold algorithm based on kernel density estimation to cope with potential changes in data patterns of small and medium-sized factories. This method avoids the problem of false positives or false negatives that may be caused by fixed thresholds by dynamically calculating the threshold.

[0015] Step 6: By combining the dynamic sparse attention mechanism mentioned in the second stage, a comprehensive anomaly score is generated for the data using the attention weights generated by the mechanism and the reconstruction error of the data;

[0016] Step 7: Determine whether the industrial control data is abnormal by combining the abnormal score and the dynamic threshold;

[0017] Furthermore, in step 1, the original data is segmented in a window form, and the first step is to calculate the preliminary alignment path: for sequence X Lmax and YLmax , at the widest time step L max Calculate the DTW distance to obtain the alignment path where ||·|| represents the Euclidean distance. The second step is path refinement: starting from the coarsest resolution L, gradually increase the resolution to L+1, and each step uses the path of the previous step as a guide to refine the alignment path. At each resolution L, the path L =Refine(path L+1 ,X L ,Y L ), where the Refine function is the process of refining the path at a higher resolution based on the low-resolution path. The third step is to calculate the final alignment path and DTW distance: At the highest resolution of the original time series, the most detailed alignment path is obtained. 0 , use this path to calculate the final DTW distance to evaluate the similarity between sequences, Among them, path (0) is the final path obtained by refining the original sequence, which can be used to evaluate the similarity between sequences. Through the above steps, the similarity matrix S is finally generated.

[0018] Furthermore, in step 2, the sample data is normalized. Then, a sliding window technique is applied, t i ={i×shift+1,i×shift+2,...,i×shift+w size Generate time series samples suitable for model training from the normalized data;

[0019] Furthermore, in step 3, an improved dynamic sparse attention mechanism is proposed. First, the window size W is determined using the autocorrelation function (ACF). size Use the determined window size to divide the feature map into regions X′; use adaptive programming to calculate the optimal top-k value k, and obtain the region-level query matrix, key matrix, and value matrix Q = X′W Q , K = X′W K ,V=X′W V , the inter-regional association graph is constructed T =Q′(K′) T , and finally perform local search attention K in the associated top-k regions g =Gather(K,I T ), V g =Gather(V,I T ), O = Attention(Q,K g ,V g ), and obtain the final score;

[0020] Furthermore, in step 4, a skip mechanism is added to VAE, and its conditional gating is The use of the latent variable z is dynamically adjusted as needed in each layer, thereby optimizing computational efficiency while keeping the model fully utilizing the latent space information.

[0021] Furthermore, in step 5, a dynamic threshold algorithm of kernel density estimation is introduced, and its formula is: Where K is the kernel function. h is the bandwidth, which controls the width of the kernel. n is the number of samples. i is the sample point, is the estimated probability density function of the anomaly score. After obtaining the density function, the probability CDF of the anomaly score being less than or equal to a certain value is obtained by integrating the PDF.

[0022] Furthermore, in step 6, a dynamic sparse attention mechanism is added to generate a total anomaly score. Among them, H t is the input of the current time step, h t-1 is the hidden state of the previous time step, α t is the attention weight, and the score(·) function is used to calculate the similarity between the current input and the migration state. The weight α is obtained t Then, the weighted anomaly score is obtained by multiplying it with the mean square error.

[0023] The positive effects of the present invention are:

[0024] The present invention firstly proposes an enhanced few-sample data preprocessing method based on kernel density guidance to solve the problem of imbalanced normal data categories; secondly, a lightweight network DSAVE is proposed, which uses an autoencoder with dynamic sparse attention mechanism and dynamic skipping mechanism to balance the accuracy and speed of the model; finally, a weighted scoring algorithm based on kernel density estimation to dynamically select thresholds and sparse attention mechanism is proposed to solve the problem of increased misjudgment rate caused by data fluctuations in production data of small and medium-sized factories, thereby improving the accuracy and reliability of anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is the structure diagram of the NK-DSAVE algorithm.

[0026] Figure 2 It is a structural diagram of the dynamic sparse attention mechanism.

[0027] Figure 3 This is the structure diagram of the VAE skip mechanism.

[0028] Figure 4 These are the different indicators and training times for various attention mechanisms.

[0029] Figure 5 This is a comparison chart of various indicators of each model. DETAILED DESCRIPTION

[0030] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0031] like Figure 1 As shown, the present invention is divided into three stages: data preprocessing, data reconstruction and dynamic threshold. The red dotted line part is the main innovative work proposed in this chapter. The first stage is the data preprocessing stage. First, the original data is segmented in the form of a window, and then the kernel density estimation method is applied to the data in the window to calculate the density score for each data point. This step aims to identify a few samples in normal data through density scores, and then increase the amount of small sample data by adding Gaussian noise to these few sample data. Next, the sample data is normalized to ensure that all subsequent operations are performed on a unified data scale. Finally, the normalized data is segmented using a sliding window technique. The second stage is the data reconstruction stage. In view of the limited computing resources of small and medium-sized factories, an improved dynamic sparse attention mechanism is first proposed to reduce the model's demand for computing resources, ensuring that the model can focus on important data features, thereby improving detection accuracy. In addition, a dynamic skip mechanism is proposed in the variational autoencoder, which accelerates the convergence speed by dynamically transferring the potential feature information to the decoder, and also improves the model's utilization efficiency of potential features. The third stage is the abnormality judgment stage. First, a dynamic threshold algorithm based on kernel density estimation is introduced. To cope with potential changes in data patterns of small and medium-sized factories. This method avoids the problem of false positives or negatives that may be caused by fixed thresholds by dynamically calculating thresholds. Then, by combining the dynamic sparse attention mechanism mentioned in the second stage, the attention weights generated by this mechanism and the reconstruction error of the data are used to generate a comprehensive anomaly score for the data. By comparing the anomaly score with the dynamic threshold for anomaly judgment, this method not only improves the sensitivity and accuracy of anomaly detection, but also provides a more reliable data monitoring solution for small and medium-sized factories. It specifically includes the following steps:

[0032] Step 1: Split the original data into windows, and then apply the kernel density estimation method to the data in the window to calculate the density score for each data point. This step aims to identify minority samples in normal data through density scores, and then increase the amount of minority sample data by adding Gaussian noise to these minority sample data.

[0033] Step 2: Normalize the sample data to ensure that all subsequent operations are performed on a unified data scale. Then use the sliding window technology to segment the normalized data.

[0034] Step 3: Data reconstruction phase. Aiming at the problem of limited computing resources in small and medium-sized factories, an improved dynamic sparse attention mechanism is proposed to reduce the model's demand for computing resources, ensuring that the model can focus on important data features, thereby improving detection accuracy.

[0035] Step 4: A dynamic skipping mechanism is proposed in the variational autoencoder, which accelerates the convergence speed by dynamically transferring the latent feature information to the decoder, and also improves the efficiency of the model in utilizing the latent features.

[0036] Step 5: Introduce a dynamic threshold algorithm based on kernel density estimation to cope with potential changes in data patterns of small and medium-sized factories. This method avoids the problem of false positives or false negatives that may be caused by fixed thresholds by dynamically calculating the threshold.

[0037] Step 6: By combining the dynamic sparse attention mechanism mentioned in the second stage, a comprehensive anomaly score is generated for the data using the attention weights generated by the mechanism and the reconstruction error of the data.

[0038] Step 7: Determine whether the industrial control data is abnormal by combining the anomaly score and the dynamic threshold.

[0039] Furthermore, in step 1, the original data is segmented in a window form, and the first step is to calculate the preliminary alignment path: for sequence X Lmax and Y Lmax , at the widest time step L max Calculate the DTW distance to obtain the alignment path where ||·|| represents the Euclidean distance. The second step is path refinement: starting from the coarsest resolution L, gradually increase the resolution to L+1, and each step uses the path of the previous step as a guide to refine the alignment path. At each resolution L, the path L =Refine(path L+1 ,X L ,Y L ), where the Refine function is the process of refining the path at a higher resolution based on the low-resolution path. The third step is to calculate the final alignment path and DTW distance: At the highest resolution of the original time series, the most detailed alignment path is obtained. 0 , use this path to calculate the final DTW distance to evaluate the similarity between sequences, Among them, path (0) is the final path obtained by refining the original sequence, which can be used to evaluate the similarity between sequences. Through the above steps, the similarity matrix S is finally generated.

[0040] Furthermore, in step 2, the sample data is normalized. Then, a sliding window technique is applied, t i ={i×shift+1,i×shift+2,...,i×shift+w size Generate time series samples suitable for model training from the normalized data;

[0041] Furthermore, in step 3, an improved dynamic sparse attention mechanism is proposed. First, the window size W is determined using the autocorrelation function (ACF). size Use the determined window size to divide the feature map into regions X′; use adaptive programming to calculate the optimal top-k value k, and obtain the region-level query matrix, key matrix, and value matrix Q = X′W Q , K = X′W K ,V=X′W V , the inter-regional association graph is constructed T =Q′(K′) T , and finally perform local search attention K in the associated top-k regions g =Gather(K,I T ), V g =Gather(V,I T ), O = Attention(Q,K g ,V g ), and obtain the final score;

[0042] Furthermore, in step 4, a skip mechanism is added to VAE, and its conditional gating is The use of the latent variable z is dynamically adjusted as needed in each layer, thereby optimizing computational efficiency while keeping the model fully utilizing the latent space information.

[0043] Furthermore, the dynamic threshold algorithm of kernel density estimation is introduced in step 5, and its formula is: Where K is the kernel function. h is the bandwidth, which controls the width of the kernel. n is the number of samples. i is the sample point, is the estimated probability density function of the anomaly score. After obtaining the density function, the probability CDF of the anomaly score being less than or equal to a certain value is obtained by integrating the PDF.

[0044] Furthermore, in step 6, a dynamic sparse attention mechanism is added to generate a total anomaly score. Among them, H t is the input of the current time step, h t-1 is the hidden state of the previous time step, α t is the attention weight, and the score(·) function is used to calculate the similarity between the current input and the migration state. The weight α is obtainedt Then, the weighted anomaly score is obtained by multiplying it with the mean square error.

Claims

1. This patent proposes a variational autoencoder anomaly detection algorithm based on a dynamic sparse attention mechanism. The method includes the following steps: Step 1: The original data is segmented in the form of a window, and then the kernel density estimation method is applied to the data in the window to calculate the density score for each data point. This step aims to identify minority samples in normal data through density scores, and then increase the amount of minority sample data by adding Gaussian noise to these minority sample data; Step 2: Normalize the sample data to ensure that all subsequent operations are performed on a unified data scale, and then use the sliding window technology to segment the normalized data; Step 3: In the data reconstruction phase, in order to address the problem of limited computing resources in small and medium-sized factories, an improved dynamic sparse attention mechanism is proposed to reduce the model's demand for computing resources, ensuring that the model can focus on important data features, thereby improving detection accuracy. Step 4: A dynamic skipping mechanism is proposed in the variational autoencoder, which accelerates the convergence speed by dynamically transferring the potential feature information to the decoder, and also improves the efficiency of the model in utilizing the potential features; Step 5: Introduce a dynamic threshold algorithm based on kernel density estimation to cope with the potential changes in the data patterns of small and medium-sized factories. This method avoids the false positive or false negative problems that may be caused by fixed thresholds by dynamically calculating the thresholds. Step 6: By combining the dynamic sparse attention mechanism mentioned in the second stage, a comprehensive anomaly score is generated for the data using the attention weights generated by the mechanism and the reconstruction error of the data; Step 7: Determine whether the industrial control data is abnormal by combining the anomaly score and the dynamic threshold.

2. According to the variational autoencoder anomaly detection algorithm based on dynamic sparse attention mechanism described in claim 1, the original data is segmented in the form of a window in step 1, and the first step is to perform preliminary alignment path calculation: for the sequence X Lmax and Y Lmax , at the widest time step L max Calculate the DTW distance to obtain the alignment path where ||·|| represents the Euclidean distance; the second step is path refinement: starting from the coarsest resolution L, the resolution is gradually increased to L+1, and each step uses the path of the previous step as a guide to refine the alignment path. At each resolution L, the path L =Refine(path L+1 ,X L ,Y L ), where the Refine function is the process of refining the path at a higher resolution based on the low-resolution path; the third step is to calculate the final alignment path and DTW distance: at the highest resolution of the original time series, the most detailed alignment path is obtained 0 , use this path to calculate the final DTW distance to evaluate the similarity between sequences. The formula is where path (0) It is the final path obtained by refining the original sequence, which can be used to evaluate the similarity between sequences. Through the above steps, the similarity matrix S is finally generated.

3. According to the variational autoencoder anomaly detection algorithm based on dynamic sparse attention mechanism described in claim 1, the sample data is normalized in step 2 Then, a sliding window technique is applied, t i ={i×shift+1,i×shift+2,...,i×shift+w size Generate time series samples suitable for model training from the normalized data.

4. According to the variational autoencoder anomaly detection algorithm based on dynamic sparse attention mechanism described in claim 1, an improved dynamic sparse attention mechanism is proposed in step 3, firstly, the window size W is determined using the autocorrelation function (ACF) size , use the determined window size to divide the feature map into regions X′; use adaptive planning to calculate the optimal top-k value k, and obtain the region-level query matrix, key matrix and value matrix Q = X′W Q , K = X′W K ,V=X′W V , the inter-regional association graph is constructed T =Q′(K′) T , and finally perform local search attention K in the associated top-k regions g =Gather(K,I T ), V g =Gather(V,I T ), O = Attention(Q,K g ,V g ) to obtain the final score.

5. According to the variational autoencoder anomaly detection algorithm based on dynamic sparse attention mechanism described in claim 1, a skip mechanism is added to VAE in step 4, and its conditional gating is The use of the latent variable z is dynamically adjusted as needed in each layer, thereby optimizing computational efficiency while keeping the model fully utilizing the latent space information.

6. According to the variational autoencoder anomaly detection algorithm based on dynamic sparse attention mechanism described in claim 1, the dynamic threshold algorithm of kernel density estimation is introduced in step 5, and its formula is: Where K is the kernel function, h is the bandwidth, which controls the width of the kernel, n is the number of samples, and x i is the sample point, is the estimated probability density function of the anomaly score. After obtaining the density function, the probability CDF of the anomaly score being less than or equal to a certain value is obtained by integrating the PDF. The formula is 7. According to the variational autoencoder anomaly detection algorithm based on dynamic sparse attention mechanism described in claim 1, the dynamic sparse attention mechanism is added in step 6 to generate a total anomaly score, in, H t is the input of the current time step, h t-1 is the hidden state of the previous time step, α t is the attention weight, sc The ore(·) function is used to calculate the similarity between the current input and the migration state, and obtain the weight α t Then, the weighted anomaly score is obtained by multiplying it with the mean square error. Determine whether the industrial control data is abnormal by combining the anomaly score and dynamic threshold.

Images