Abnormity detection algorithm based on global attention mechanism and comparative learning

Abstract

In order to solve the problem that an existing anomaly detection model of a large factory is difficult to capture complex time sequence features and inter-dimension relation features at the same time, the invention designs an anomaly detection algorithm GCAVE based on a global attention mechanism and comparative learning. Firstly, a global self-attention mechanism is designed to fully capture time sequence characteristics in a single dimension and relation characteristics among different dimensions of multi-parameter time sequence data; and then an anomaly detection mechanism combining confrontation and comparative learning is designed to improve the distinguishing capability of the model for abnormal data, and meanwhile, the problem of posterior collapse possibly occurring in the training process of the model is prevented. And finally, designing a time sequence integrated gradient anomaly interpretation algorithm to position an abnormal sensor.

Description

Technical Field

[0001] The present invention relates to industrial control safety, and in particular to an anomaly detection algorithm based on a global attention mechanism and contrastive learning. Background Art

[0002] With the widespread popularity of the Internet and the rapid advancement of information technology, large factories in China have received close attention and strong support from the government. The expansion of factory scale and the complexity of production processes make it difficult for traditional monitoring and management methods to meet the needs. Any abnormal neglect or delay may cause serious consequences. Therefore, the development of anomaly detection algorithms with high accuracy and anomaly interpretation capabilities is of vital importance to ensure production safety and maintain economic benefits.

[0003] Based on the above situation, researchers began to explore spatiotemporal feature extraction methods suitable for multivariate time series industrial control data in complex scenarios. This method improves the accuracy of model detection in complex industrial environments by improving the deep learning network structure to prevent serious industrial control safety accidents. At present, deep learning technology is widely used in the field of anomaly detection in complex scenarios. Currently, algorithms that can achieve high anomaly detection accuracy are divided into three categories: algorithms based on time features, algorithms based on spatial features, and algorithms based on spatiotemporal features. They each show significant advantages in dealing with specific problems, but also have their own limitations. The first type of method can effectively capture the internal dependencies of time series data, but ignores the correlation between different dimensions, which limits its anomaly detection accuracy in processing highly correlated sensor data in industrial control environments. For example, the OC4Seq framework proposed by Wang Z et al. (Wang Z, Chen Z, Ni J, et al. Multi-scale one-class recurrent neural networks for discrete event sequence anomaly detection [C] / / Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining. 2021: 3726-3734) combines the advantages of one-class classification and recurrent neural networks to learn the representation of sequences at global and local scales to detect anomalies. Wu D (Wu D, Jiang Z, Xie X, et al. LSTM learning with Bayesian and Gaussian processing for anomaly detection in industrial IoT [J]. IEEE Transactions on Industrial Informatics, 2019, 16 (8): 5244-5253.) et al. use long short-term memory networks to capture long-term dependencies in time series, and use the classification performance of the Gaussian naive Bayes model to classify the predicted data to detect abnormal data. Although the second type of method can better capture correlation features, its ability to extract time features is weak, which affects the overall accuracy.For example, Zhou Y (Zhou Y, Liang X, Zhang W, et al. VAE-based deep SVDD for anomaly detection [J]. Neurocomputing, 2021, 453: 131-140.) et al. first extracted features through VAE, and then used support vector data description to learn a hypersphere to distinguish normal and abnormal data. Han C (Han C, Rundo L, Murao K, et al. MADGAN: Unsupervised medical anomaly detection GAN using multiple adjacent brain MRI slice reconstruction [J]. BMC bioinformatics, 2021, 22 (2): 1-20.) et al. combined generative adversarial networks and self-attention mechanisms to detect anomalies. The third category, such as Yu J et al. (

[44] Yu J, Song Y, Tang D, et al. Telemetry data-based spacecraft anomaly detection with spatial–temporal generative adversarial networks [J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-9.), uses a hybrid model of convolutional neural networks and long short-term memory networks to extract spatial and temporal features from the observed data respectively. This type of method uses a hybrid model to extract and fuse features from two aspects, showing good detection results. However, there is still a lot of room for improvement in its accuracy, and there are problems with insufficient stability and convergence, which affect the detection accuracy.

[0004] The above analysis shows that although the research on anomaly detection for multidimensional time series data has achieved certain results, the existing models are still insufficient in fully capturing the time series and inter-dimensional features of complex data. Given the rich data features generated by large factories, there is still room for improvement in the accuracy of anomaly detection using integrated models. At the same time, the generative model may suffer from the posterior collapse problem due to its overly strong decoding ability, which will lead to overfitting of the model and thus weaken the generalization ability of the model. In addition, existing models often lack the ability to explain anomalies.

[0005] In order to solve the above problems, this chapter proposes an anomaly detection algorithm GCAVE based on global attention mechanism and contrastive learning. First, this chapter uses the spectral residual method to amplify the difference between the potential distribution of normal and abnormal data. Secondly, in view of the problem that the existing model fails to simultaneously extract the temporal features within the dimension and the relationship features between dimensions, this chapter proposes a global attention mechanism to effectively process the data within and between blocks, giving the model the ability to identify anomalies at the local and global levels. In addition, adversarial learning is used to solve the posterior collapse, and the model's ability to distinguish anomalies is improved through contrastive loss. Finally, in view of the lack of anomaly explainability of existing models, this chapter proposes an improved time series integrated gradient anomaly interpretation algorithm. Summary of the invention

[0006] The purpose of this invention is to design a variational autoencoder anomaly detection algorithm based on global attention mechanism and contrastive adversarial learning to solve the problem that the existing anomaly detection model of large factories is difficult to capture complex time series features and inter-dimensional relationship features at the same time.

[0007] The object of the present invention is achieved in this way:

[0008] First, the spectral residual algorithm is used to perform Fourier transform on the data to enhance the difference between normal and abnormal samples. Secondly, adversarial learning and contrastive learning mechanisms are designed to solve the posterior collapse and improve the model's ability to distinguish abnormal data. At the same time, a global attention mechanism is designed to improve the model's ability to capture spatiotemporal features of multi-parameter time series data. Finally, an improved time series integrated gradient algorithm is designed to locate abnormal sensors.

[0009] The specific method is as follows:

[0010] An anomaly detection algorithm based on global attention mechanism and contrastive learning includes the following steps:

[0011] Step 1: Data preprocessing based on spectral residual and normalization. In order to solve the problem that some abnormal samples have similar potential distributions to normal samples, which leads to small model reconstruction errors, this chapter uses the spectral residual method in the data preprocessing stage to amplify the distribution difference between abnormal samples and normal samples to improve the reconstruction error of abnormal samples.

[0012] Step 2: Build a network based on the global attention mechanism for data reconstruction. Considering the inability of existing models to capture spatiotemporal features at the same time, a global attention mechanism is built. This mechanism performs dual-perspective processing on the blocks within and between blocks of time series data, which enables the model to capture not only the local and global features of the data, but also the spatial features between sensors.

[0013] Step 3: Strengthen the ability of the data reconstruction network to capture abnormal data. To address the problem that the generative model is not capable of identifying abnormal data that is similar to the normal data distribution, we build an adversarial learning and contrastive learning mechanism to improve the ability of the encoder and decoder to identify abnormalities.

[0014] Step 4: Build an improved time series integrated gradient anomaly detection algorithm to locate abnormal sensors for abnormal data. By calculating the relative contribution of each sensor in the anomaly score, the potential problem sensors can be accurately located, thereby improving the efficiency of operation and maintenance personnel in handling anomalies;

[0015] Furthermore, in step 1, the data preprocessing based on spectral residual and normalization is to perform Fourier transform on the signal in the window, then calculate the amplitude, calculate the phase, apply filtering to the amplitude using Gaussian kernel, apply filtering to the amplitude log spectrum using Gaussian kernel, calculate the spectral residual, perform inverse Fourier transform with the adjusted amplitude and original phase, and add the processed result to the result set. Then the enhanced data is normalized. Finally, the sliding window is used to generate time window samples suitable for model training. Specifically, for the original training data of N dimensions, dimi represents the i-th dimension. First, a sliding window is applied to each sensor signal. Xi By setting the window size to w_size and sliding it in the time dimension with a step size of 1, we can obtain several multidimensional time window samples of size N*w_size. For any time window sample t input into the GCAVE model Ni ,have

[0016] Furthermore, in step 2, a network based on the global attention mechanism is constructed. First, the features output from the GRU module are processed in blocks. Specifically, the output multivariate time series data is X∈R T×d , where T is the time step. d is the feature dimension, i.e. the number of different sensors. By defining the window size W and step size S, the data is further divided into a series of time blocks. Each window can be further represented as X i =X[t i :t i +W], where t is the starting time step of the window. Then, in the feature extraction stage, the intra-block self-attention mechanism and the inter-block self-attention mechanism are introduced. The main purpose of the intra-block self-attention mechanism is to capture the temporal dependencies within each time block and reveal the short-term relationships within the sequence. i ∈R W×d Construct query Q, key K and value V, map the original features in the time block to the new feature space, and then use the attention score calculated by the similarity between Q and K, after scaling factor Normalization is performed to enhance the learning stability of the model. At the same time, the inter-block self-attention mechanism aims to understand the long-term dependencies between different time windows, thereby capturing the global relationship of the sequence. By treating the entire sequence as a whole and applying a multi-head self-attention mechanism to strengthen the connection between different time blocks, the overall structure of the time series can be understood from a macro perspective. First, the set of all blocks is represented as B∈R N×d , where N is the number of blocks divided, and d is the feature dimension of each block. For the inter-block self-attention mechanism, it is also necessary to generate query Q, key K, and value V, but unlike the inter-block self-attention mechanism, they are generated based on the set B of the entire sequence block. Finally, the two feature maps output by the global time series attention mechanism are fused. The fused feature map contains both local features and global features, which is more suitable for anomaly detection tasks in industrial control environments. First, feature dimensionality reduction is performed to extract key information. 1×1 convolution is applied to the intra-block and inter-block feature maps respectively, F′ 内 =Conv 1×1 (F 内 ), F′ 间 =Conv 1×1 (F 间 ) Next, the processed intra-block and inter-block feature maps are concatenated, F 融合 =Concat(F′ 内 , F′ 间 ) In this way, F 融合 It contains information provided by the self-attention mechanism within and between blocks, and the feature dimension is expanded to 2d. Finally, in order to retain part of the original input features and promote gradient flow, the original input features are added to the fusion features through residual connections to obtain the final feature map, F 残差 =F 原始 +F 残差输入 Finally, the feature map with expanded dimensions is reduced in dimension through 1×1 convolution to restore it to the initial feature dimension dF' 最终 =Conv 1×1 (F 整合 );

[0017] Furthermore, the ability of the data reconstruction network to capture abnormal data is enhanced in step 3. First, the encoder network maps the multi-parameter time series data to a representation z in a latent space, and its output is the parameters of the latent representation, usually the mean μ and variance σ 2 Here, z is derived from a Gaussian distribution N(μ,σ 2 ) is sampled, Among them, ε is the random noise sampled from the standard normal distribution N(0,I), represents element-wise multiplication. The mean and variance of z are modified by applying a fully connected layer linear transformation and a nonlinear activation function. The virtual network first applies a linear transformation to z, which is formulated as follows: linear =W T ·z+b T Where W T is the parameter weight matrix of the virtual network, b T is the bias vector, z linear is the result after applying linear transformation. Then the virtual network further processes the result after linear transformation by applying nonlinear activation function, thereby increasing the expressive power of the model. The formula is z'=f(z linear ) Combining these two steps, the abnormal potential representation z' generated by the virtual network can be expressed as: z'=f(W T ·z+b T ) where f(·) is a nonlinear activation function. In this section, the sigmoid function is used. The abnormal latent representation z′ generated by the virtual network and the normal latent representation z generated by the encoder are transmitted to the decoder layer together. The decoder attempts to decode these two similar latent representations z and z′ respectively to generate normal reconstructed data W and abnormal reconstructed data W'.

[0018] Furthermore, the ability of the data reconstruction network to capture abnormal data is strengthened in step 3. In order to enhance the anomaly detection ability of the model and prevent posterior collapse, contrast loss and adversarial loss are introduced. The reconstruction-based anomaly detection framework usually has a similar objective function, as shown below:

[0019]

[0020] Where G(·) represents the reconstruction model and R(G) is the regularization term. The objective function proposed in this chapter adds two additional regularization terms on this basis: adversarial loss L D and contrast loss L E The objective function is as follows:

[0021] L VAE =L MSE (W,W')+L KLD (μ,σ)

[0022]

[0023] Among them, L VAE is the loss function of the variational autoencoder, which is mainly composed of two main terms: mean square error loss L MSE and KL divergence loss L KLD .L MSE The loss evaluates the difference between the reconstruction error W' output by the model and the target, which is the original data W.KLD The loss measures the distribution of the latent representation N(μ,σ 2 ) and the standard normal distribution. It quantifies the difference between the two probability distributions. Adversarial loss L D This part also consists of two terms. The first term is the KL divergence in the latent space, which is used to quantify the abnormal latent representation generated by the virtual network T. and the target distribution N(z;μ,σ 2 ). The second term is a correction term, which means that only when the mean square error loss is greater than the threshold m x α is a hyperparameter that adds a linear growth to the correction term, allowing the model to converge faster. The role of this correction term is to avoid overfitting of normal data and focus the model's reconstruction ability on distinguishing normal and abnormal potential representations. Adversarial loss can improve the decoder's ability to distinguish normal and abnormal data during model training, and constrain the decoder's generation ability to avoid posterior collapse. Contrastive loss L E This part is also composed of two terms. The first term is the normal potential representation N(μ,σ) output by the encoder 2 ) with another normal latent representation The second term is a correction term that is only valid when the KL divergence between the normal latent representation and the abnormal latent representation is less than the threshold m. z β is a hyperparameter that adds a linear growth to the correction term, allowing the model to converge faster. The role of this correction term is to label the model to form a more dispersed representation in the latent space, making it easier to identify abnormal data. The addition of contrast loss can improve the encoder's ability to distinguish normal and abnormal vectors in the latent space, thereby increasing the difference between normal and abnormal data reconstruction in the subsequent decoding stage.

[0024] Furthermore, in step 4, an improved time-series integrated gradient anomaly detection algorithm is constructed to locate abnormal sensors for abnormal data. The algorithm is mainly used to explain the prediction decisions of deep neural networks. Based on this method, this design quantifies the contribution of each sensor to the anomaly score by comprehensively analyzing the gradient changes from the normal state to the abnormal state, thereby locating the abnormal sensor. IG integrates the gradient changes from the baseline state to the actual observed state, revealing the impact of each dimension on the model's judgment results. Since the neutral baseline state is crucial to understanding how the model identifies specific attributes, this section improves the IG algorithm to meet the special needs of anomaly detection. The IG formula is as follows:

[0025] Where F(·) represents the deep network function, x t ∈R M Represents an abnormal entity, x tb∈R M is the corresponding neutral baseline entity, M is the number of dimensions of each entity, and α is the definition of x tb With x t IG calculates x tb and x t The deviation D(x tb ,x t ) to identify x t Based on the above formula, the GCAVE anomaly model is set to F, and by comparing the abnormal state x t and the baseline state x reconstructed by the model tb The difference can be used to m (x tb ,x t ) to identify the contribution of each dimension. m (x tb ,x t ) formula is Among them, x tb is the reconstructed baseline state, x t is the abnormal state detected, and are the values ​​of these two states in the mth dimension respectively. In this way, the TIG algorithm can not only identify anomalies, but also provide detailed explanations about why the anomalies occur;

[0026] The positive effects of the present invention are:

[0027] Aiming at the problem that the existing anomaly detection models of large factories are difficult to capture complex time series features and inter-dimensional relationship features at the same time, the present invention designs an anomaly detection algorithm GCAVE based on global attention mechanism and contrastive learning. Firstly, a global self-attention mechanism is designed to fully capture the time series features within a single dimension of multi-parameter time series data and the relationship features between different dimensions. Subsequently, an anomaly detection mechanism combining adversarial and contrastive learning is designed to improve the model's ability to distinguish abnormal data, while preventing the posterior collapse problem that may occur in the model during the training process. Finally, a time-series integrated gradient anomaly interpretation algorithm is designed to locate abnormal sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is the structural diagram of the anomaly detection algorithm based on global attention mechanism and contrastive learning.

[0029] Figure 2 It is a contrastive learning generative network framework.

[0030] Figure 3 It is the abnormal contribution diagram of each sensor.

[0031] Figure 4is a scatter plot of feature vectors in the latent space.

[0032] Figure 5 It is the decoding effect of each model decoder under the influence of different noises. DETAILED DESCRIPTION

[0033] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0034] like Figure 1 As shown, the present invention is mainly divided into three stages: data preprocessing, data reconstruction, and anomaly determination and location. The first stage is data preprocessing based on spectral residual and normalization. In view of the problem that some abnormal samples have similar potential distributions to normal samples, resulting in a small model reconstruction error, this chapter uses the spectral residual method in the data preprocessing stage to amplify the distribution difference between abnormal samples and normal samples to improve the reconstruction error of abnormal samples. In the second stage, an anomaly detection algorithm based on global attention mechanism and contrastive learning is proposed for data reconstruction. In view of the problem that the generative model is not capable of identifying abnormal data with similar distribution to normal data, it is proposed to improve the ability of encoder and decoder to identify anomalies based on adversarial learning and contrastive learning mechanisms. In addition, considering the insufficient ability of existing models in capturing spatiotemporal features at the same time, a global attention mechanism is proposed. This mechanism can not only enable the model to capture the local and global features of the data, but also the spatial features between sensors by performing dual-perspective processing on the blocks within and between blocks of time series data. In the third stage, an improved time series integrated gradient anomaly detection algorithm is proposed to locate abnormal sensors for abnormal data. The algorithm calculates the relative contribution of each sensor in the anomaly score to accurately locate potential problem sensors, thereby improving the efficiency of operation and maintenance personnel in handling anomalies. The specific steps include:

[0035] Step 1: Data preprocessing based on spectral residual and normalization. In order to solve the problem that some abnormal samples have similar potential distributions to normal samples, which leads to small model reconstruction errors, this chapter uses the spectral residual method in the data preprocessing stage to amplify the distribution difference between abnormal samples and normal samples to improve the reconstruction error of abnormal samples.

[0036] Step 2: Build a network based on the global attention mechanism for data reconstruction. Considering the inability of existing models to capture spatiotemporal features at the same time, a global attention mechanism is built. This mechanism performs dual-perspective processing on the blocks within and between blocks of time series data, which enables the model to capture not only the local and global features of the data, but also the spatial features between sensors.

[0037] Step 3: Strengthen the ability of the data reconstruction network to capture abnormal data. To address the problem that the generative model is not capable of identifying abnormal data that is similar to the normal data distribution, we build an adversarial learning and contrastive learning mechanism to improve the ability of the encoder and decoder to identify abnormalities.

[0038] Step 4: Build an improved time series integrated gradient anomaly detection algorithm to locate abnormal sensors for abnormal data. By calculating the relative contribution of each sensor in the anomaly score, the potential problem sensors can be accurately located, thereby improving the efficiency of operation and maintenance personnel in handling anomalies;

[0039] Furthermore, in step 1, the data preprocessing based on spectral residual and normalization is to perform Fourier transform on the signal in the window, then calculate the amplitude, calculate the phase, apply filtering to the amplitude using Gaussian kernel, apply filtering to the amplitude log spectrum using Gaussian kernel, calculate the spectral residual, perform inverse Fourier transform with the adjusted amplitude and original phase, and add the processed result to the result set. Then the enhanced data is normalized. Finally, the sliding window is used to generate time window samples suitable for model training. Specifically, for the original training data of N dimensions, dimi represents the i-th dimension. First, a sliding window is applied to each sensor signal. Xi By setting the window size to w_size and sliding it in the time dimension with a step size of 1, we can obtain several multidimensional time window samples of size N*w_size. For any time window sample t input into the GCAVE model Ni ,have

[0040] Furthermore, in step 2, a network based on the global attention mechanism is constructed. First, the features output from the GRU module are processed in blocks. Specifically, the output multivariate time series data is X∈R T×d , where T is the time step. d is the feature dimension, i.e. the number of different sensors. By defining the window size W and step size S, the data is further divided into a series of time blocks. Each window can be further represented as X i =X[t i :t i +W], where t is the starting time step of the window. Then, in the feature extraction stage, the intra-block self-attention mechanism and the inter-block self-attention mechanism are introduced. The main purpose of the intra-block self-attention mechanism is to capture the temporal dependencies within each time block and reveal the short-term relationships within the sequence. i ∈R W×d Construct query Q, key K and value V, map the original features in the time block to the new feature space, and then use the attention score calculated by the similarity between Q and K, after scaling factor Normalization is performed to enhance the learning stability of the model. At the same time, the inter-block self-attention mechanism aims to understand the long-term dependencies between different time windows, thereby capturing the global relationship of the sequence. By treating the entire sequence as a whole and applying a multi-head self-attention mechanism to strengthen the connection between different time blocks, the overall structure of the time series can be understood from a macro perspective. First, the set of all blocks is represented as B∈R N×d , where N is the number of blocks divided, and d is the feature dimension of each block. For the inter-block self-attention mechanism, it is also necessary to generate query Q, key K, and value V, but unlike the inter-block self-attention mechanism, they are generated based on the set B of the entire sequence block. Finally, the two feature maps output by the global time series attention mechanism are fused. The fused feature map contains both local features and global features, which is more suitable for anomaly detection tasks in industrial control environments. First, feature dimensionality reduction is performed to extract key information. 1×1 convolution is applied to the intra-block and inter-block feature maps respectively, F′ 内 =Conv 1×1 (F 内 ), F′ 间 =Conv 1×1 (F 间 ) Next, the processed intra-block and inter-block feature maps are concatenated, F 融合 =Concat(F′ 内 , F′ 间 ) In this way, F 融合 It contains information provided by the self-attention mechanism within and between blocks, and the feature dimension is expanded to 2d. Finally, in order to retain part of the original input features and promote gradient flow, the original input features are added to the fusion features through residual connections to obtain the final feature map, F 残差 =F 原始 +F 残差输入 Finally, the feature map with expanded dimensions is reduced in dimension through 1×1 convolution to restore it to the initial feature dimension dF' 最终 =Conv 1×1 (F 整合 );

[0041] Furthermore, the ability of the data reconstruction network to capture abnormal data is enhanced in step 3. First, the encoder network maps the multi-parameter time series data to a representation z in a latent space, and its output is the parameters of the latent representation, usually the mean μ and variance σ 2 Here, z is derived from a Gaussian distribution N(μ,σ 2 ) is sampled, Among them, ε is the random noise sampled from the standard normal distribution N(0,I), represents element-wise multiplication. The mean and variance of z are modified by applying a fully connected layer linear transformation and a nonlinear activation function. The virtual network first applies a linear transformation to z, which is formulated as follows: linear =W T ·z+b T Where W T is the parameter weight matrix of the virtual network, b T is the bias vector, z linear is the result after applying linear transformation. Then the virtual network further processes the result after linear transformation by applying nonlinear activation function, thereby increasing the expressive power of the model. The formula is z'=f(z linear ) Combining these two steps, the abnormal potential representation z' generated by the virtual network can be expressed as: z'=f(W T ·z+b T ) where f(·) is a nonlinear activation function. In this section, the sigmoid function is used. The abnormal latent representation z′ generated by the virtual network and the normal latent representation z generated by the encoder are transmitted to the decoder layer together. The decoder attempts to decode these two similar latent representations z and z′ respectively to generate normal reconstructed data W and abnormal reconstructed data W'.

[0042] Furthermore, the ability of the data reconstruction network to capture abnormal data is strengthened in step 3. In order to enhance the anomaly detection ability of the model and prevent posterior collapse, contrast loss and adversarial loss are introduced. The reconstruction-based anomaly detection framework usually has a similar objective function, as shown below:

[0043]

[0044] Where G(·) represents the reconstruction model and R(G) is the regularization term. The objective function proposed in this chapter adds two additional regularization terms on this basis: adversarial loss L D and contrast loss L E The objective function is as follows:

[0045] L VAE =L MSE (W,W')+L KLD (μ,σ)

[0046]

[0047] Among them, L VAE is the loss function of the variational autoencoder, which is mainly composed of two main terms: mean square error loss L MSE and KL divergence loss L KLD .L MSE The loss evaluates the difference between the reconstruction error W' output by the model and the target, which is the original data W.KLD The loss measures the distribution of the latent representation N(μ,σ 2 ) and the standard normal distribution. It quantifies the difference between the two probability distributions. Adversarial loss L D This part also consists of two terms. The first term is the KL divergence in the latent space, which is used to quantify the abnormal latent representation generated by the virtual network T. and the target distribution N(z;μ,σ 2 ). The second term is a correction term, which means that only when the mean square error loss is greater than the threshold m x α is a hyperparameter that adds a linear growth to the correction term, allowing the model to converge faster. The role of this correction term is to avoid overfitting of normal data and focus the model's reconstruction ability on distinguishing normal and abnormal potential representations. Adversarial loss can improve the decoder's ability to distinguish normal and abnormal data during model training, and constrain the decoder's generation ability to avoid posterior collapse. Contrastive loss L E This part is also composed of two terms. The first term is the normal potential representation N(μ,σ) output by the encoder 2 ) with another normal latent representation The second term is a correction term that is only valid when the KL divergence between the normal latent representation and the abnormal latent representation is less than the threshold m. z β is a hyperparameter that adds a linear growth to the correction term, allowing the model to converge faster. The role of this correction term is to label the model to form a more dispersed representation in the latent space, making it easier to identify abnormal data. The addition of contrast loss can improve the encoder's ability to distinguish normal and abnormal vectors in the latent space, thereby increasing the difference between normal and abnormal data reconstruction in the subsequent decoding stage.

[0048] Furthermore, in step 4, an improved time-series integrated gradient anomaly detection algorithm is constructed to locate abnormal sensors for abnormal data. The algorithm is mainly used to explain the prediction decisions of deep neural networks. Based on this method, this design quantifies the contribution of each sensor to the anomaly score by comprehensively analyzing the gradient changes from the normal state to the abnormal state, thereby locating the abnormal sensor. IG integrates the gradient changes from the baseline state to the actual observed state, revealing the impact of each dimension on the model's judgment results. Since the neutral baseline state is crucial to understanding how the model identifies specific attributes, this section improves the IG algorithm to meet the special needs of anomaly detection. The IG formula is as follows:

[0049] Where F(·) represents the deep network function, x t ∈R M Represents an abnormal entity, x tb∈R M is the corresponding neutral baseline entity, M is the number of dimensions of each entity, and α is the definition of x tb With x t IG calculates x tb and x t The deviation D(x tb ,x t ) to identify x t Based on the above formula, the GCAVE anomaly model is set to F, and by comparing the abnormal state x t and the baseline state x reconstructed by the model tb The difference can be used to m (x tb ,x t ) to identify the contribution of each dimension. m (x tb ,x t ) formula is Among them, x tb is the reconstructed baseline state, x t is the abnormal state detected, and are the values ​​of these two states in the mth dimension. In this way, the TIG algorithm can not only identify anomalies, but also provide detailed explanations about why the anomalies occurred.

[0050] Figure 3 This is the abnormal contribution diagram of each sensor, which proves the effectiveness of the time series integrated gradient abnormal interpretation algorithm. The analysis results of the algorithm provide maintenance personnel with the key information needed to quickly diagnose and troubleshoot system abnormalities, thereby helping to improve the troubleshooting efficiency of operation and maintenance personnel. Figure 4 It can be seen that the encoder of the method of the present invention performs better in distinguishing normal and abnormal vectors in the latent space. This enhanced recognition ability increases the difference between normal and abnormal data reconstruction in the subsequent decoding stage, thereby improving the performance of the model in the anomaly detection task. This result proves the effectiveness of contrastive learning in optimizing anomaly detection algorithms. Figure 5 It is proved that the present invention shows the best robustness in various noise environments.

Claims

1. An anomaly detection algorithm based on global attention mechanism and contrastive learning, the method comprising the following steps: Step 1: Data preprocessing based on spectral residual and normalization; In order to solve the problem that some abnormal samples have similar potential distributions to normal samples, which leads to small model reconstruction errors, this chapter uses the spectral residual method in the data preprocessing stage to amplify the distribution difference between abnormal samples and normal samples to improve the reconstruction error of abnormal samples; Step 2: Build a network based on the global attention mechanism for data reconstruction. Considering the inability of existing models to capture spatiotemporal features at the same time, a global attention mechanism is built. This mechanism performs dual-perspective processing on the blocks within and between blocks of time series data, which enables the model to capture not only the local and global features of the data, but also the spatial features between sensors. Step 3: Strengthen the ability of the data reconstruction network to capture abnormal data; in order to solve the problem that the generative model is not able to identify abnormal data with similar distribution to normal data, we build an adversarial learning and contrastive learning mechanism to improve the ability of the encoder and decoder to identify abnormal data; Step 4: Build an improved time series integrated gradient anomaly detection algorithm to locate abnormal sensors for abnormal data; by calculating the relative contribution of each sensor in the anomaly score, the potential problem sensors can be accurately located, thereby improving the efficiency of operation and maintenance personnel in handling anomalies.

2. According to the anomaly detection algorithm based on global attention mechanism and contrastive learning described in claim 1, in step 1, the data preprocessing based on spectral residual and normalization is performed, the signal in the window is subjected to Fourier transform, and then the amplitude is calculated, the phase is calculated, the amplitude is filtered using a Gaussian kernel, the amplitude log spectrum is filtered using a Gaussian kernel, the spectral residual is calculated, the adjusted amplitude and the original phase are used for inverse Fourier transform, and the processed result is added to the result set; and then the enhanced data is normalized. Finally, the sliding window is used to generate time window samples suitable for model training. Specifically, for the original training data of N dimensions, dimi represents the i-th dimension; first apply a sliding window to each sensor signal Xi , by setting the window size to w_size and sliding it in the time dimension with a step size of 1, we can get several multidimensional time window samples of size N*w_size; for any time window sample t input into the GCAVE model Ni ,have 3. According to the anomaly detection algorithm based on global attention mechanism and contrastive learning described in claim 1, in step 2, a network based on global attention mechanism is constructed; first, the features output from the GRU module are processed in blocks; specifically, the output multivariate time series data is X∈R T×d , where T is the time step; d is the feature dimension, that is, the number of different sensors; by defining the window size W and step size S, the data is further divided into a series of time blocks, and each window can be further represented as X i =X[t i :t i +W], where t is the starting time step of the window; then in the feature extraction stage, the intra-block self-attention mechanism and the inter-block self-attention mechanism are introduced; the main purpose of the intra-block self-attention mechanism is to capture the temporal dependencies within each time block and reveal the short-term relationship within the sequence; by i ∈R W×d Construct query Q, key K and value V, map the original features in the time block to the new feature space, and then use the attention score calculated by the similarity between Q and K, after scaling factor Normalization is performed to enhance the learning stability of the model. At the same time, the inter-block self-attention mechanism aims to understand the long-term dependencies between different time windows, thereby capturing the global relationship of the sequence. By considering the entire sequence as a whole and applying the multi-head self-attention mechanism to strengthen the connection between different time blocks, the overall structure of the time series can be understood from a macro perspective. First, the set of all blocks is represented as B∈R N×d , where N is the number of blocks divided, and d is the feature dimension of each block; for the inter-block self-attention mechanism, it is also necessary to generate query Q, key K and value V, but unlike the inter-block self-attention mechanism, they are generated based on the set B of the entire sequence block. Finally, the two feature maps output by the global time series attention mechanism are fused. The fused feature map contains both local features and global features, which is more suitable for anomaly detection tasks in industrial control environments. First, feature dimensionality reduction is performed to extract key information. 1×1 convolution is applied to the intra-block and inter-block feature maps respectively, F′ 内 =Conv 1×1 (F 内 ), F′ 间 =Conv 1×1 (F 间 ) Next, the processed intra-block and inter-block feature maps are concatenated, F 融合 =Concat(F′ 内 , F′ 间 ) In this way, F 融合 It contains information provided by the self-attention mechanism within and between blocks, and the feature dimension is expanded to 2d; finally, in order to retain part of the original input features and promote gradient flow, the original input features are added to the fusion features through residual connections to obtain the final feature map, F 残差 =F 原始 +F 残差输入 Finally, the feature map with expanded dimensions is reduced in dimension through 1×1 convolution to restore it to the initial feature dimension dF' 最终 =Conv 1×1 (F 整合 ).

4. According to the anomaly detection algorithm based on global attention mechanism and contrastive learning described in claim 1, the ability of the data reconstruction network to capture abnormal data is enhanced in step 3; first, the encoder network maps the multi-parameter time series data to a representation z in a latent space, and its output is the parameters of the latent representation, usually the mean μ and the variance σ 2 Here, z is an estimate of 2 ) is sampled, in, ε is the random noise sampled from the standard normal distribution N(0,I), represents element-wise multiplication; the mean and variance of z are modified by applying linear changes and nonlinear activation functions of the fully connected layer; the virtual network first applies a linear transformation to z, which is formulated as follows, z linear =W T ·z+b T Where W T is the parameter weight matrix of the virtual network, b T is the bias vector, z linear is the result after applying linear transformation; then the virtual network applies nonlinear activation function to the result after linear transformation to further process it, thereby increasing the expressive power of the model; the formula is z'=f(z linear ) Combining these two steps, the abnormal potential representation z' generated by the virtual network can be expressed as: z'=f(W T ·z+b T )in, f(·) is a nonlinear activation function. In this section, the sigmoid function is used. The abnormal latent representation z′ generated by the virtual network and the normal latent representation z generated by the encoder are transmitted to the decoder layer together. The decoder attempts to decode these two similar latent representations z and z′ respectively to generate normal reconstructed data W and abnormal reconstructed data W'.

5. According to the anomaly detection algorithm based on global attention mechanism and contrastive learning described in claim 1, the ability of the data reconstruction network to capture abnormal data is strengthened in step 3; in order to enhance the anomaly detection ability of the model and prevent posterior collapse, contrast loss and adversarial loss are introduced. The reconstruction-based anomaly detection framework usually has a similar objective function, and the formula is as follows: where G(·) represents the reconstruction model and R(G) is the regularization term. The objective function proposed in this chapter adds two additional regularization terms on this basis: adversarial loss L D and contrast loss L E ; The objective function is as follows: L VAE =L MSE (W,W')+L KLD (m,s) in, L VAE is the loss function of the variational autoencoder, which is mainly composed of two main terms: mean square error loss L MSE and KL divergence loss L KLD ; L MSE The loss evaluates the difference between the reconstruction error W' output by the model and the target, that is, the original data W; L KLD The loss measures the distribution of the latent representation N(μ,σ 2 ) and the standard normal distribution; it quantifies the difference between the two probability distributions; the adversarial loss L D This part also consists of two terms; the first term is the KL divergence in the latent space, which is used to quantify the abnormal latent representation generated by the virtual network T and the target distribution N(z;μ,σ 2 ) between KL divergence; the second term is a correction term, which means that only when the mean square error loss is greater than the threshold m x α is a hyperparameter that adds a linear growth to the correction term, so that the model converges faster. The role of this correction term is to avoid overfitting of normal data and focus the model's reconstruction ability on distinguishing normal and abnormal potential representations. The adversarial loss can improve the decoder's ability to distinguish normal and abnormal data during model training, and constrain the decoder's generation ability to avoid posterior collapse. The contrast loss L E This part also consists of two terms; the first term is the normal potential representation N(μ,σ) output by the encoder 2 ) with another normal latent representation The second term is a correction term that is only valid when the KL divergence between the normal latent representation and the abnormal latent representation is less than the threshold m. z β is a hyperparameter that adds a linear growth to the correction term, allowing the model to converge faster. The role of this correction term is to label the model to form a more dispersed representation in the latent space, making it easier to identify abnormal data. The addition of contrast loss can improve the encoder's ability to distinguish between normal and abnormal vectors in the latent space, thereby increasing the difference between normal and abnormal data reconstruction in the subsequent decoding stage.

6. According to the anomaly detection algorithm based on global attention mechanism and contrastive learning described in claim 1, an improved time series integrated gradient anomaly detection algorithm is constructed in step 4 to locate abnormal sensors for abnormal data; the algorithm is mainly used to explain the prediction decision of deep neural network. Based on this method, this design quantifies the contribution of each sensor to the abnormal score by comprehensively analyzing the gradient change from normal state to abnormal state, and realizes the location of abnormal sensors; IG integrates the gradient change from baseline state to actual observation state, revealing the influence of each dimension on the model judgment result; since the neutral baseline state is crucial to understanding how the model identifies specific attributes, this section improves the IG algorithm to meet the special needs of anomaly detection; the IG formula is as follows: in, F(·) represents the deep network function, x t ∈R M Represents an abnormal entity, x tb ∈R M is the corresponding neutral baseline entity, M is the number of dimensions of each entity, and α is the definition of x tb With x t The smooth curve of the relationship between IG and x tb and x t The deviation D(x tb ,x t ) to identify x t the contribution of each part; Based on the above formula, the GCAVE anomaly model is set to F, and by comparing the abnormal state x t and the baseline state x reconstructed by the model tb The difference can be used to m (x tb ,x t ) to identify the contribution of each dimension; its D m (x tb ,x t ) formula is Among them, x tb is the reconstructed baseline state, x t is the abnormal state detected, and are the values ​​of these two states in the mth dimension respectively; in this way, the TIG algorithm can not only identify anomalies but also provide detailed explanations about why the anomalies occurred.

Images