Anomaly detection method based on global attention mechanism and contrast learning

Abstract

In view to the problem that the existing abnormality detection model of a large factory is difficult to capture complex time sequence features and relationship features between dimensions simultaneously, the abnormality detection algorithm GCAVE based on global attention mechanism and contrast learning is designed. Firstly, the global self-attention mechanism is designed to fully capture the time sequence features in a single dimension and the relationship features between different dimensions of multi-parameter time sequence data. Then, the abnormality detection mechanism combining adversarial and contrast learning is designed to improve the ability of the model to distinguish abnormal data, and to prevent the posterior collapse problem of the model in the training process. Finally, the time sequence integrated gradient abnormality explanation algorithm is designed to locate the abnormal sensor.

Description

TECHNICAL FIELD

[0001] The application relates to industrial control safety, in particular to an anomaly detection method based on a global attention mechanism and contrast learning. BACKGROUND

[0002] With the widespread popularity of the Internet and the rapid progress of information technology, China's large factories have received close attention and strong support from the government. In 2015, in order to promote the combination of industrial automation and information technology and improve the technological innovation ability and the intelligent level of production, the Chinese government put forward the "Made in China 2025" strategy. This plan has received a positive response, and many large factories have developed rapidly due to government support and occupy a more prominent position in the national industrial economy. According to the data of China's State Statistics Bureau in 2020, large industrial enterprises contributed nearly half of the total national industrial profits, highlighting their key position in the national economy. However, the rapid development of these large factories also brings new challenges, especially in terms of production safety and efficiency. The expansion of factory size and the complexity of the production process make it difficult for traditional monitoring and management methods to meet the needs, and any neglect or delay of abnormalities can cause serious consequences. Therefore, developing an anomaly detection method with high precision and anomaly explanation ability is of great significance to ensure production safety and maintain economic benefits.

[0003] Based on the above status, researchers began to explore the spatio-temporal feature extraction method suitable for complex scenarios of multivariate time series industrial control data. This method improves the accuracy of model detection in complex industrial environments by improving the structure of deep learning network to prevent serious industrial control safety accidents. At present, in the field of anomaly detection in complex scenarios, deep learning technology is widely used, and the current methods that can achieve high accuracy of anomaly detection are divided into three categories: time feature-based methods, space feature-based methods, and spatio-temporal feature-based methods. Each of them has shown significant advantages in dealing with specific problems, but also has its own limitations. The first method can effectively capture the internal dependence of time series data, but it ignores the correlation between different dimensions, which limits its accuracy in detecting anomalies in industrial control environments with high correlation sensor data. For example, Wang Z et al. (Wang Z, Chen Z, Ni J, et al. Multi-scale one-class recurrent neural networks for discrete event sequence anomaly detection [C] / / Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining. 2021:3726-3734) proposed the OC4Seq framework, which combines the advantages of one-class classification and recurrent neural networks to learn the representation of sequences at global and local scales to detect anomalies. WU D (Wu D, Jiang Z, Xie X, et al. LSTM learning with Bayesian and Gaussian processing for anomaly detection in industrial IoT [J]. IEEE Transactions on Industrial Informatics, 2019, 16(8):5244-5253.) et al. captured long-term dependencies in time series through long short-term memory networks, and used the classification performance of the Gaussian naive Bayes model to classify the predicted data to detect abnormal data. The second method can better capture correlation features, but the ability to extract time features is weak, which affects the overall accuracy.Zhou Y (Zhou Y, Liang X, Zhang W, et al. VAE-based deep SVDD for anomaly detection [J]. Neurocomputing, 2021, 453: 131-140.) et al. first extract features by VAE, and then learn a hypersphere using support vector data description to distinguish normal and abnormal data. Han C (Han C, Rundo L, Murao K, et al. MADGAN: Unsupervised medical anomaly detection GAN using multiple adjacent brain MRI slice reconstruction [J]. BMC bioinformatics, 2021, 22(2): 1-20.) et al. combined generative adversarial network and self-attention mechanism to detect anomalies, etc. The third type such as Yu J et al. (

[44] Yu J, Song Y, Tang D, et al. Telemetry data-based spacecraft anomaly detection with spatial-temporal generative adversarial networks [J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-9.) uses a hybrid model of convolutional neural network and long short-term memory network to extract spatial and temporal features from observation data respectively. This kind of method uses a hybrid model to extract and fuse features from two aspects, which shows good detection effect. But its accuracy has a lot of room for improvement, and there are problems of instability and convergence, which affect the detection accuracy.

[0004] The above analysis shows that although the research on anomaly detection of multi-dimensional time series data has achieved certain results, the existing models still have deficiencies in fully capturing the time series and dimensional features of complex data. Given the rich data features generated by large factories, the current integrated model still has room for improvement in improving the accuracy of anomaly detection. At the same time, the generative model may have a posterior collapse problem due to its strong decoding ability, which will lead to overfitting of the model, thereby weakening the generalization ability of the model. In addition, the existing model often lacks the ability to explain anomalies.

[0005] To solve the above problems, this chapter proposes an anomaly detection method GCAVE based on global attention mechanism and contrastive learning. First, this chapter uses the spectral residual method to amplify the difference between the potential distribution of normal and abnormal data. Second, to solve the problem that existing models cannot extract both the time features within the dimension and the relationship features between the dimensions, this chapter proposes a global attention mechanism, which effectively processes the data within and between blocks, and gives the model the ability to identify anomalies at both local and global levels. In addition, adversarial learning is used to solve the posterior collapse, and the contrastive loss is used to improve the model's ability to distinguish anomalies. Finally, to address the lack of anomaly explainability in existing models, this chapter proposes an improved time series integrated gradient anomaly explanation method. SUMMARY

[0006] The purpose of the present application is to address the problem that existing anomaly detection models for large factories cannot simultaneously capture complex time series features and inter-dimensional relationship features. A variational autoencoder anomaly detection method based on global attention mechanism and contrastive adversarial learning is designed

[0007] The purpose of the present application is achieved as follows:

[0008] First, the spectral residual method is used to perform Fourier transform on the data to enhance the difference between normal and abnormal samples. Second, an adversarial learning and contrastive learning mechanism is designed to solve the posterior collapse and improve the model's ability to distinguish abnormal data. At the same time, a global attention mechanism is designed to improve the model's ability to capture spatio-temporal features of multi-parameter time series data. Finally, an improved time series integrated gradient method is designed to locate abnormal sensors.

[0009] The specific mode is as follows:

[0010] An anomaly detection method based on global attention mechanism and contrastive learning includes the following steps:

[0011] Step 1, data preprocessing based on spectral residual and normalization. To address the problem that some abnormal samples have similar potential distribution to normal samples, resulting in small reconstruction error of the model, this chapter uses the spectral residual method to amplify the distribution difference between abnormal samples and normal samples in the data preprocessing stage to improve the reconstruction error of abnormal samples.

[0012] Step 2, construct a global attention mechanism network for data reconstruction. Considering the insufficient ability of existing models to capture spatio-temporal features, a global attention mechanism is constructed. This mechanism processes the data within and between blocks from two perspectives, enabling the model to capture both local and global features, as well as spatial features between sensors.

[0013] Step 3, strengthen the ability of data reconstruction network to capture abnormal data. In order to solve the problem that the generated model is not good at identifying abnormal data similar to normal data distribution, an improved method based on adversarial learning and contrastive learning mechanism is constructed to improve the ability of encoder and decoder to identify abnormal data.

[0014] Step 4, construct an improved time series integrated gradient anomaly detection method to locate abnormal sensors for abnormal data. By calculating the relative contribution of each sensor in the abnormal score, the precise positioning of the potential problem sensor is realized, so as to improve the efficiency of operation and maintenance personnel in handling abnormal data;

[0015] Further, in step 1, the data preprocessing based on spectral residual and normalization, the window signal is subjected to Fourier transform, then the amplitude is calculated, the phase is calculated, the amplitude is filtered using Gaussian kernel, the amplitude logarithmic spectrum is filtered using Gaussian kernel, the spectral residual is calculated, the inverse Fourier transform is performed using the adjusted amplitude and the original phase, and the processed result is added to the result set. Then the enhanced data is normalized Finally, the sliding window is used to generate time window samples suitable for model training. Specifically, for N-dimensional original training data, dim i represents the i-th dimension. First, the sliding window is applied, and each sensor signal X i is obtained by setting the window size as w_size and sliding in the time dimension with a step size of 1, thereby obtaining a plurality of multi-dimensional time window samples with a size of N*w_size. For any time window sample t Ni input into the GCAVE model, there are

[0016] Further, in step 2, a global attention mechanism network is constructed. First, the features output from the GRU module are processed in blocks. Specifically, the output multivariate time series data is X∈R T×d , where T is the time step, and d is the feature dimension, i.e. the number of different sensors. By defining the window size W and the step size S, the data is further divided into a series of time blocks, and each window can be further represented as X i = X[t i :t i +W], where t is the starting time step of the window. Then in the feature extraction stage, the intra-block self-attention mechanism and the inter-block self-attention mechanism are introduced. The intra-block self-attention mechanism mainly aims to capture the time series dependence within each time block and reveal the short-term relationship within the sequence. By constructing query Q, key K and value V from X i ∈R W×d , the original features within the time block are mapped to a new feature space, and then the attention score calculated by the similarity of Q and K is scaled by a scaling factor Normalization is performed to enhance the learning stability of the model, while the inter-block self-attention mechanism aims to understand the long-term dependencies between different time windows, thereby capturing the global relationship of the sequence. By regarding the entire sequence as a whole and applying the multi-head self-attention mechanism to strengthen the connection between different time blocks, the overall structure of the time series can be understood from a macro perspective. First, the set of all blocks is represented as B ∈ R N×d , where N is the number of blocks divided, and d is the feature dimension of each block. For the inter-block self-attention mechanism, it is also necessary to generate queries Q, keys K, and values V, but unlike the inter-block self-attention mechanism, they are generated based on the entire sequence block set B. Finally, the two feature maps output by the global time series attention mechanism are fused, and the feature map obtained by fusion contains both local features and global features, which is more suitable for anomaly detection tasks in industrial control environments. First, feature dimensionality reduction is performed to extract key information, and 1x1 convolution is applied to the intra-block and inter-block feature maps, respectively, F' 内 = Conv 1×1 (F 内 ), F' 间 = Conv 1×1 (F 间 ) Next, the processed intra-block and inter-block feature maps are feature-stitched, F 融合 = Concat(F' 内 , F' 间 ) In this way, F 融合 contains information from the intra-block and inter-block self-attention mechanisms, and the feature dimension is expanded to 2d. Finally, in order to preserve a portion of the original input features and facilitate gradient flow, the original input features are added to the fused features through a residual connection to obtain the final feature map, F 残差 = F 原始 + F 残差输入 Finally, the feature map with expanded dimensions is reduced in dimension by 1x1 convolution to restore the initial feature dimension d F' 最终 = Conv 1×1 (F 整合 ) ;

[0017] Further, the step 3 strengthens the ability of the data reconstruction network to capture abnormal data. First, the encoder network maps the multi-parameter time series data to a representation z in a latent space, and its output is the parameters of the latent representation, usually the mean μ and variance σ 2 . Here, z is sampled from a Gaussian distribution N(μ,σ 2 ), where ε is random noise sampled from a standard normal distribution N(0, I), represents element-wise multiplication. The mean and variance of z are modified by applying a fully connected layer linear transformation and a nonlinear activation function. The virtual network first applies a linear transformation to z, which is formulated as follows, z linear = W T · z + b T where W T is the parameter weight matrix of the virtual network, b T is the bias vector, and z linear is the result after applying the linear transformation. Then the virtual network applies a nonlinear activation function to the result of the linear transformation to further process, thus increasing the expressive power of the model. The formula is z' = f(z linear ). Combining these two steps, the abnormal latent representation z' generated by the virtual network can be represented as: z' = f(W T · z + b T ) where f(·) is the nonlinear activation function, and the sigmoid function is used in this section. For the abnormal latent representation z' generated by the virtual network and the normal latent representation z generated by the encoder, they are transmitted together to the decoder layer, and the decoder tries to decode these two similar hidden representations z and z' respectively to generate normal reconstructed data W and abnormal reconstructed data W'.

[0018] Further, the step 3 strengthens the ability of the data reconstruction network to capture abnormal data. To enhance the model's anomaly detection ability and prevent posterior collapse, contrastive loss and adversarial loss are introduced, and the anomaly detection framework based on reconstruction generally has a similar objective function, which is shown as follows:

[0019] L = ||x - G(x)||2+ R(G) (4-19) where G(·) represents the reconstruction model, and R(G) is the regularization term. The objective function proposed in this chapter adds two additional regularization terms: adversarial loss L D and contrastive loss L E . The objective function is shown as follows:

[0020] L VAE = L MSE (W, W') + L KLD (μ, σ)

[0021]

[0022] where L VAE is the loss function of the variational autoencoder, which mainly consists of two main terms: mean square error loss L MSE and KL divergence loss L KLD . L MSE loss evaluates the difference between the reconstructed error W' output by the model and the target, i.e., the original data W. L KLDThe loss measures the difference between the distribution of latent representations N(μ,σ 2 ) and the standard normal distribution. It quantifies the difference between two probability distributions. The adversarial loss L D This part is also composed of two terms. The first term is the KL divergence in the latent space, which is used to quantify the KL divergence between the abnormal latent representation generated by the virtual network T and the target distribution N(z; μ,σ 2 ). The second term is a modification term, which only works when the mean square error loss is greater than the threshold m x . a is a hyperparameter that adds a linear growth to the modification term, so that the model converges faster. The role of this modification term is to avoid overfitting to normal data, so that the reconstruction ability of the model is focused on distinguishing normal and abnormal latent representations. The adversarial loss can improve the ability of the decoder to distinguish between normal and abnormal data in model training, and constrain the generation ability of the decoder to avoid posterior collapse. The contrastive loss L E This part is also composed of two terms. The first term is the KL divergence between the normal latent representation N(μ,σ 2 ) output by the encoder and another normal latent representation . The second term is a modification term, which only works when the KL divergence between the normal latent representation and the abnormal latent representation is less than the threshold m z . β is a hyperparameter that adds a linear growth to the modification term, so that the model converges faster. The role of this modification term is to mark the model to form more dispersed representations in the latent space, so that it is easier to identify abnormal data. The addition of the contrastive loss can improve the ability of the encoder to distinguish between normal and abnormal vectors in the latent space, so that the difference between the reconstruction of normal and abnormal data can be increased in the subsequent decoding stage.

[0023] Further, the improved time series ensemble gradient anomaly detection method constructed in step 4 is used to locate the abnormal sensor of the abnormal data. The method is mainly used to explain the prediction decision of the deep neural network, and the design is based on this method to comprehensively analyze the gradient change from the normal state to the abnormal state, quantify the contribution degree of each sensor to the abnormal score, and realize the positioning of the abnormal sensor. IG reveals the influence of each dimension on the model decision result by integrating the gradient change from the baseline state to the actual observation state. Since the neutral baseline state is crucial to understanding how the model identifies specific attributes, this section improves the IG method to meet the special needs of anomaly detection. The IG formula is as follows:

[0024] Where F(·) represents the deep network function, x t ∈R M represents the abnormal entity, and x tb ∈RM For the corresponding neutral baseline entity, M is the number of dimensions for each entity, and α is the dimension defined by x. tb With x t A smooth curve relating the two. IG is calculated by x. tb and x t The deviation between D(x) tb ,x t To identify x t The contribution of each part. Based on the above formula, the GCAVE anomaly model is set to F, and the contribution of each part is compared with the anomaly state x. t Compared with the baseline state x reconstructed by the model tb The difference can be used with D m (x tb ,x t This is used to identify contributions in each dimension. Where, x tb It is the reconstructed baseline state, x t It is an abnormal state detected. and These are the values ​​of the two states in the m-th dimension, respectively. In this way, the TIG method can not only identify anomalies but also provide detailed explanations of the causes of the anomalies.

[0025] The positive effects of this invention are:

[0026] This invention addresses the challenge of existing anomaly detection models in large factories failing to simultaneously capture complex temporal features and inter-dimensional relationships. It proposes GCAVE, an anomaly detection method based on global attention and contrastive learning. First, a global self-attention mechanism is designed to fully capture the temporal features within a single dimension of multi-parameter time-series data, as well as the relationships between different dimensions. Then, an anomaly detection mechanism combining adversarial and contrastive learning is designed to improve the model's ability to distinguish anomaly data while preventing posterior collapse that may occur during training. Finally, a temporal integrated gradient anomaly interpretation method is designed to locate anomalies in the sensor data. Attached Figure Description

[0027] Figure 1 This is a structural diagram of an anomaly detection method based on global attention mechanism and contrastive learning.

[0028] Figure 2 It is a contrastive learning generative network framework.

[0029] Figure 3 This is a graph showing the abnormal contribution of each sensor.

[0030] Figure 4 It is a scatter plot of latent space feature vectors.

[0031] Figure 5This represents the decoding performance of each model decoder under different noise conditions. Detailed Implementation

[0032] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0033] like Figure 1 As shown, this invention mainly consists of three stages: data preprocessing, data reconstruction, and anomaly detection and localization. The first stage involves data preprocessing based on spectral residuals and normalization. Addressing the issue of small model reconstruction errors due to the similar potential distributions of some anomalous samples and normal samples, this chapter uses the spectral residual method in the data preprocessing stage to amplify the distribution differences between anomalous and normal samples, thereby improving the reconstruction error of anomalous samples. The second stage proposes an anomaly detection method based on a global attention mechanism and contrastive learning for data reconstruction. Addressing the insufficient ability of generative models to identify anomalous data with distributions similar to normal data, this stage proposes an adversarial learning and contrastive learning mechanism to enhance the encoder and decoder's ability to identify anomalies. Furthermore, considering the insufficient ability of existing models to simultaneously capture spatiotemporal features, a global attention mechanism is proposed. This mechanism, through dual-perspective processing of time-series data within and between blocks, enables the model to capture not only local and global features of the data but also spatial features between sensors. The third stage proposes an improved temporal integrated gradient anomaly detection method for anomaly sensor localization. This method calculates the relative contribution of each sensor to the anomaly score, enabling precise location of potentially problematic sensors and thus improving the efficiency of maintenance personnel in handling anomalies. The specific steps include:

[0034] Step 1: Data Preprocessing Based on Spectral Residuals and Normalization. To address the issue of small model reconstruction errors due to the similar potential distributions of some outlier samples and normal samples, this chapter uses the spectral residual method in the data preprocessing stage to amplify the distributional differences between outlier and normal samples, thereby improving the reconstruction error of outlier samples.

[0035] Step 2: Construct a network based on a global attention mechanism for data reconstruction. Considering the limitations of existing models in simultaneously capturing spatiotemporal features, a global attention mechanism is constructed. This mechanism processes the time series data from both intra-block and inter-block perspectives, enabling the model to capture not only local and global features of the data but also spatial features between sensors.

[0036] Step 3: Enhance the data reconstruction network's ability to capture anomalous data. To address the issue of insufficient ability of generative models to identify anomalous data similar to normal data distributions, an adversarial learning and contrastive learning mechanism is constructed to improve the encoder and decoder's ability to identify anomalies.

[0037] Step 4, an improved time series integrated gradient anomaly detection method is constructed to perform abnormal sensor positioning on abnormal data. By calculating the relative contribution of each sensor in the anomaly score, the precise positioning of the potential problem sensor is realized, thereby improving the efficiency of the operation and maintenance personnel in handling anomalies;

[0038] Further, in step 1, based on the spectral residual and normalized data preprocessing, the signal in the window is subjected to Fourier transformation, then the amplitude is calculated, the phase is calculated, the amplitude is filtered using a Gaussian kernel, the amplitude log spectrum is filtered using a Gaussian kernel, the spectral residual is calculated, the adjusted amplitude and the original phase are subjected to inverse Fourier transformation, and the processed result is added to the result set. Then the enhanced data is normalized Finally, a sliding window is used to generate a time window sample suitable for model training. Specifically, for N-dimensional original training data, dim i represents the i-th dimension. First, a sliding window is applied to each sensor signal X i , by setting the window size to w_size and sliding in the time dimension with a step size of 1, a number of multi-dimensional time window samples with a size of N*w_size are obtained. For any time window sample t Ni input to the GCAVE model, there is

[0039] Further, in step 2, a global attention mechanism network is constructed. First, the features output from the GRU module are processed in blocks. Specifically, the output multivariate time series data is X∈R T×d , where T is the time step, and d is the feature dimension, i.e., the number of different sensors. By defining the window size W and the step size S, the data is further divided into a series of time blocks, and each window can be further represented as X i = X[t i :t i +W], where t is the starting time step of the window. Then in the feature extraction stage, the intra-block self-attention mechanism and the inter-block self-attention mechanism are introduced. The intra-block self-attention mechanism mainly aims to capture the time series dependency within each time block and reveal the short-term relationship within the sequence. By constructing X i ∈R W×d , the query Q, the key K and the value V are constructed to map the original features within the time block to a new feature space. Then, the attention score calculated by the similarity of Q and K is scaled by the scaling factor Normalization is performed to enhance the learning stability of the model, while the inter-block self-attention mechanism aims to understand the long-term dependencies between different time windows, thereby capturing the global relationship of the sequence. By regarding the entire sequence as a whole and applying the multi-head self-attention mechanism to strengthen the connection between different time blocks, the overall structure of the time series can be understood from a macro perspective. First, the set of all blocks is represented as B ∈ R N×d , where N is the number of blocks divided, and d is the feature dimension of each block. For the inter-block self-attention mechanism, it is also necessary to generate queries Q, keys K, and values V, but unlike the inter-block self-attention mechanism, they are generated based on the entire sequence block set B. Finally, the two feature maps output by the global time series attention mechanism are fused, and the feature map obtained by fusion contains both local features and global features, which is more suitable for anomaly detection tasks in industrial control environments. First, feature dimensionality reduction is performed to extract key information, and 1x1 convolution is applied to the intra-block and inter-block feature maps, respectively, F' 内 = Conv 1×1 (F 内 ), F' 间 = Conv 1×1 (F 间 ) Next, the processed intra-block and inter-block feature maps are concatenated, F 融合 = Concat(F' 内 , F' 间 ) In this way, F 融合 contains information from the intra-block and inter-block self-attention mechanisms, and the feature dimension is expanded to 2d. Finally, in order to preserve a portion of the original input features and facilitate gradient flow, the original input features are added to the fused features through a residual connection to obtain the final feature map, F 残差 = F 原始 + F 残差输入 Finally, the feature map with expanded dimensions is reduced in dimension through 1x1 convolution to restore the initial feature dimension d F' 最终 = Conv 1×1 (F 整合 ) ;

[0040] Further, the step 3 strengthens the ability of the data reconstruction network to capture abnormal data. First, the encoder network maps the multi-parameter time series data to a representation z in a latent space, and its output is the parameters of the latent representation, usually the estimates of the mean μ and variance σ 2 Here, z is sampled from a Gaussian distribution N(μ,σ 2 ), where ε is random noise sampled from a standard normal distribution N(0, I), represents element-wise multiplication. The mean and variance of z are modified by applying a fully connected layer linear transformation and a nonlinear activation function. The virtual network first applies a linear transformation to z, which is formulated as follows, z linear = W T ·z + b T where W T is the parameter weight matrix of the virtual network, b T is the bias vector, and z linear is the result after applying the linear transformation. Then the virtual network applies a nonlinear activation function to the result of the linear transformation to further process, thus increasing the expressive power of the model. The formula is z' = f(z linear ). Combining these two steps, the abnormal latent representation z' generated by the virtual network can be represented as: z' = f(W T ·z + b T ) where f(·) is the nonlinear activation function, and the sigmoid function is used in this section. For the abnormal latent representation z' generated by the virtual network and the normal latent representation z generated by the encoder, they are transmitted together to the decoder layer, and the decoder tries to decode these two similar hidden representations z and z' respectively to generate normal reconstructed data W and abnormal reconstructed data W'.

[0041] Further, the step 3 strengthens the ability of the data reconstruction network to capture abnormal data. To enhance the model's anomaly detection ability and prevent posterior collapse, contrastive loss and adversarial loss are introduced, and the anomaly detection framework based on reconstruction generally has a similar objective function, which is shown as follows:

[0042] L = ||x - G(x)||2+ R(G) (4-19) where G(·) represents the reconstruction model, and R(G) is the regularization term. The objective function proposed in this chapter adds two additional regularization terms: adversarial loss L D and contrastive loss L E . The objective function is shown as follows:

[0043] L VAE = L MSE (W, W') + L KLD (μ, σ)

[0044]

[0045] where L VAE is the loss function of the variational autoencoder, which mainly consists of two main terms: mean square error loss L MSE and KL divergence loss L KLD . L MSE loss evaluates the difference between the reconstructed error W' of the model output and the target, i.e., the original data W. L KLDThe loss measures the difference between the distribution of latent representations N(μ,σ 2 ) and the standard normal distribution. It quantifies the difference between two probability distributions. The adversarial loss L D This part is also composed of two terms. The first term is the KL divergence in the latent space, which is used to quantify the KL divergence between the abnormal latent representation generated by the virtual network T and the target distribution N(z; μ,σ 2 ). The second term is a modification term, which only works when the mean square error loss is greater than the threshold m x . a is a hyperparameter that adds a linear growth to the modification term, so that the model converges faster. The role of this modification term is to avoid overfitting to normal data, so that the reconstruction ability of the model is focused on distinguishing normal and abnormal latent representations. The adversarial loss can improve the ability of the decoder to distinguish between normal and abnormal data during model training, and constrain the generation ability of the decoder to avoid posterior collapse. The contrastive loss L E This part is also composed of two terms. The first term is the KL divergence between the normal latent representation N(μ,σ 2 ) output by the encoder and another normal latent representation . The second term is a modification term, which only works when the KL divergence between the normal latent representation and the abnormal latent representation is less than the threshold m z . β is a hyperparameter that adds a linear growth to the modification term, so that the model converges faster. The role of this modification term is to mark the model to form more dispersed representations in the latent space, so that it is easier to identify abnormal data. The addition of the contrastive loss can improve the ability of the encoder to distinguish between normal and abnormal vectors in the latent space, so that the difference between the reconstruction of normal and abnormal data can be increased in the subsequent decoding stage.

[0046] Further, the improved time series ensemble gradient anomaly detection method is constructed in step 4 to locate the abnormal sensor of the abnormal data. The method is mainly used to explain the prediction decision of the deep neural network, and the design is based on this method to comprehensively analyze the gradient change from the normal state to the abnormal state, quantify the contribution degree of each sensor to the abnormal score, and realize the positioning of the abnormal sensor. IG reveals the influence of each dimension on the model decision result by integrating the gradient change from the baseline state to the actual observation state. Since the neutral baseline state is crucial to understanding how the model identifies specific attributes, this section improves the IG method to meet the special needs of anomaly detection. The IG formula is as follows:

[0047] Where F(·) represents the deep network function, x t ∈R M represents an abnormal entity, and x tb RM For the corresponding neutral baseline entity, M is the number of dimensions of each entity, a is a smooth curve that defines the relationship between x tb With x t . IG identifies the contribution of each part of x tb By calculating the deviation D(x t , x tb ) between x t and x t . Based on the above formula, set the GCAVE anomaly model as F, by comparing the difference between the abnormal state x t and the baseline state x tb reconstructed by the model, the contribution of each dimension can be identified using D m (x tb , x t ). Its D m (x tb , x t ) formula is Where x tb is the reconstructed baseline state, x t is the detected abnormal state, And are the values of the two states in the mth dimension. In this way, the TIG method can not only identify anomalies, but also provide detailed explanations about the causes of the anomalies

[0048] Figure 3 The sensor anomaly contribution map proves the effectiveness of the time series integrated gradient anomaly explanation method. The analysis results of the method provide the key information required for the maintenance personnel to quickly diagnose and troubleshoot system anomalies, thereby helping to improve the troubleshooting efficiency of the maintenance personnel. From Figure 4 It can be seen that the encoder of the method of the present application performs better in distinguishing normal and abnormal vectors in the latent space. This enhanced recognition ability increases the difference between the reconstruction of normal and abnormal data in the subsequent decoding stage, thereby improving the performance of the model in the anomaly detection task. The results prove the effectiveness of contrastive learning in optimizing the anomaly detection method. Figure 5 It is proved that the present application shows the best robustness in various noise environments.

Claims

1. An anomaly detection method based on global attention mechanism and contrastive learning, the method comprising the following steps: Step 1, data preprocessing based on spectral residual and normalization; In order to solve the problem that the model reconstruction error is small because the potential distribution of some abnormal samples and normal samples is similar, the spectral residual method is used to enlarge the distribution difference between abnormal samples and normal samples in the data preprocessing stage, so as to improve the reconstruction error of abnormal samples; Step 2, constructing a global attention mechanism network for data reconstruction; considering the insufficient ability of existing models in capturing spatio-temporal features, a global attention mechanism is constructed; this mechanism can capture local and global features of data and spatial features between sensors by processing the data in and between blocks from two perspectives; Step 3, strengthening the ability of data reconstruction network to capture abnormal data; in order to solve the problem that the generated model is insufficient in identifying abnormal data similar to normal data distribution, an improved method based on adversarial learning and contrastive learning mechanism is constructed to improve the ability of encoder and decoder to identify abnormalities; Step 4, constructing an improved time series integrated gradient anomaly detection method to locate abnormal sensors for abnormal data; by calculating the relative contribution of each sensor to the abnormal score, the precise positioning of the potential problem sensor is realized, thereby improving the efficiency of operation and maintenance personnel in handling abnormalities; The step 2 constructs a global attention mechanism network; first, the features output from the GRU module are processed by block; specifically, the output multivariate time series data is X ∈ R T×d , where T is the time step; d is the feature dimension, that is, the number of different sensors; by defining the window size W and the step size S, the data is further segmented into a series of time blocks, and each window can be further represented as X i = X[t i :t i +W], where t is the starting time step of the window; Then the feature extraction stage, the introduction of the block within the self-attention mechanism and block self-attention mechanism; the block within the self-attention mechanism is mainly to capture the time dependence within each time block, reveal the short-term relationship within the sequence; through the X i ∈R W×d The construction of the query Q, key K and value V, the original features within the time block are mapped into a new feature space, then the attention score calculated by the similarity of Q and K is scaled by the scaling factor Normalization, enhance the learning stability of the model, at the same time, the block self-attention mechanism aims to understand the long-term dependence between different time windows, so as to capture the global relationship of the sequence; by regarding the entire sequence as a whole and applying multi-head self-attention mechanism to strengthen the connection between different time blocks, the overall structure of the time series can be understood from a macro perspective; first, the set of all blocks is represented as B∈R N×d , where N is the number of blocks divided, and d is the feature dimension of each block; for the block self-attention mechanism, it also needs to generate query Q, key K and value V, but unlike the block self-attention mechanism, they are generated based on the entire sequence block set B, finally the two feature maps output by the global time series attention mechanism are fused, the feature map obtained by fusion contains both local features and global features, which is more suitable for abnormal detection tasks in industrial control environment, first, feature dimension reduction is performed to extract key information, 1×1 convolution is applied to the block within and block between feature maps respectively, F 内 =Conv 1×1 (F 内 ), F 间 =Conv 1×1 (F 间 )Next, the processed block within and block between feature maps are spliced, F 融合 =Concat(F' 内 , F' 间 )In this way, F 融合 contains information from the block within and block between self-attention mechanisms, and the feature dimension is expanded to 2d; finally, in order to preserve part of the original input features and promote gradient flow, the original input features are added to the fused features through residual connection, and the final feature map F 残差 is obtained, F 原始 +F 残差输入 Finally, the feature map with expanded dimension is reduced in dimension through 1×1 convolution, and the initial feature dimension d is restored F' 最终 =Conv 1×1 (F 整合 ); An improved time series integrated gradient anomaly detection method is constructed to locate abnormal sensors for abnormal data; the method is mainly used to explain the prediction decision of deep neural network, and the design is based on this method to comprehensively analyze the gradient change from normal state to abnormal state, quantify the contribution of each sensor to the abnormal score, and realize the positioning of abnormal sensors; IG reveals the influence of each dimension on the model decision result by integrating the gradient change from the baseline state to the actual observation state; since the neutral baseline state is crucial to understanding how the model identifies specific attributes, the IG method is improved to meet the special needs of anomaly detection; the IG formula is as follows: Where F(·) represents the deep network function, x t ∈R M Indicates an abnormal entity, x tb ∈R M For the corresponding neutral baseline entity, M is the number of dimensions for each entity, and a is the definition of x. tb With x t A smooth curve representing the relationship between them; IG is calculated by x tb and x t The deviation between D(x) tb ,x t To identify x t The contributions of each part; based on the above formula, the GCAVE anomaly model is set to F, by comparing the anomaly state x t Compared with the baseline state x reconstructed by the model tb The difference can be used with D m (x tb ,x t To identify contributions in each dimension; its D m (x tb ,x t The formula is Where, x tb It is the reconstructed baseline state, x t It is an abnormal state detected. and These are the values ​​of the two states in the m-th dimension, respectively; in this way, the TIG method can not only identify anomalies, but also provide a detailed explanation of the reasons for the anomalies.

2. The anomaly detection method based on global attention mechanism and contrastive learning according to claim 1, wherein in step 1, the data preprocessing based on spectral residual and normalization, the signal in the window is subjected to Fourier transformation, then the amplitude is calculated, the phase is calculated, the amplitude is filtered using a Gaussian kernel, the amplitude logarithmic spectrum is filtered using a Gaussian kernel, the spectral residual is calculated, the inverse Fourier transformation is performed using the adjusted amplitude and the original phase, and the processed result is added to the result set; then the normalized processing is performed on the enhanced data Finally, a time window sample suitable for model training is generated by using a sliding window. Specifically, for the original training data of N dimensions, dim i represents the i-th dimension; first, a sliding window is applied, and each sensor signal X i is obtained by setting the window size to w_size and sliding in the time dimension with a step size of 1, thereby obtaining a plurality of multi-dimensional time window samples with a size of N*w_size; for any time window sample t Ni input to the GCAVE model, there are 3. The anomaly detection method based on global attention mechanism and contrastive learning according to claim 1, in step 3, the ability of data reconstruction network to capture abnormal data is strengthened; in order to enhance the anomaly detection ability of the model and prevent posterior collapse, contrastive loss and adversarial loss are introduced, and the anomaly detection framework based on reconstruction usually has a similar objective function, the formula is as follows: L = ||x - G(x)||2+ R(G) (4-19) where G(·) represents the reconstruction model, and R(G) is the regularization term; the proposed objective function adds two additional regularization terms: the adversarial loss L D and the contrastive loss L E ; the objective function is shown as follows: L VAE = L MSE (W,W')+L KLD (μ,σ) wherein L VAE The loss function of the variational autoencoder consists of two main parts: the mean square error loss L MSE and the KL divergence loss L KLD ;L MSE The loss evaluates the reconstruction error of the model output W' and the target, i.e., the original data W;L KLD The loss measures the difference between the latent representation distribution N(μ,σ 2 ) and the standard normal distribution; it quantifies the difference between two probability distributions; the adversarial loss L D This part also consists of two terms; the first term is the KL divergence in the latent space, which is used to quantify the KL divergence between the abnormal latent representation generated by the virtual network T and the target distribution N(z;μ,σ 2 ); the second term is a correction term, which only works when the mean square error loss is greater than the threshold m x ; α is a hyperparameter that adds a linear growth to the correction term, so that the model converges faster; the role of this correction term is to avoid overfitting to normal data, so that the reconstruction ability of the model is focused on distinguishing normal and abnormal latent representations; the adversarial loss can improve the ability of the decoder to distinguish normal and abnormal data in the model training, and constrain the generation ability of the decoder to avoid posterior collapse; the contrastive loss L E This part also consists of two terms; the first term is the KL divergence between the normal latent representation N(μ,σ 2 ) output by the encoder and another normal latent representation ; the second term is a correction term, which only works when the KL divergence between the normal latent representation and the abnormal latent representation is less than the threshold m z ; β is a hyperparameter that adds a linear growth to the correction term, so that the model converges faster; the role of this correction term is to mark the model to form more dispersed representations in the latent space, so that it is easier to identify abnormal data; the addition of the contrastive loss can improve the encoder's ability to distinguish normal and abnormal vectors in the latent space, so that the difference between the reconstruction of normal and abnormal data can be increased in the subsequent decoding stage.

Images