Multivariable time sequence anomaly detection method based on abnormal attention and gated storage
Through the multivariate timing abnormality detection method combined with abnormal attention and gated storage module, the problems of low detection accuracy and excessive time in the prior art are solved, and more efficient abnormality detection is achieved, suitable for complex time series data.
Patent Information
- Application Number
- CN202411007354.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-12
AI Technical Summary
The existing multivariate timing abnormality detection methods based on reconstruction are prone to low detection accuracy and long time due to excessive generalization, and it is difficult to effectively capture the correlation differences of complex time series.
Using a multivariate timing anomaly detection method based on abnormal attention and gated storage, the abnormal attention module and gated storage module are combined to improve the reconstruction process, and the learnable Gaussian core and locally sensitive hash attention mechanism are used to combine it. The detection criteria of two-dimensional deviation and multi-stage training enhance the generalization ability and detection accuracy of the model.
It improves the accuracy and speed of multivariate timing anomaly detection, enhances the robustness and stability of the model, and can more effectively capture correlation differences, and is suitable for complex multivariate time series data.
Smart Images

Figure CN118940180A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of time series anomaly detection, and in particular to a multivariate time series anomaly detection method based on abnormal attention and gated storage. Background Art
[0002] Currently, with the popularity of big data and complex systems, multivariate time series data plays an increasingly important role in various fields, such as finance, industrial production, health monitoring, etc. However, there may be various anomalies in these data, such as failures, fraud, or changes in health status. Traditional time series anomaly detection methods perform poorly in dealing with anomalies in these complex multivariate time series data.
[0003] As deep learning has made significant progress in the past few years, it is increasingly capable of learning representations of complex time series, such as multidimensional data with spatial and temporal features. In deep learning anomaly detection, neural networks are used to learn feature representations or anomaly scores to detect anomalies. Many deep learning anomaly detection models have been developed and applied in a variety of practical applications, with performance significantly higher than traditional time series anomaly detection models.
[0004] Reconstruction-based methods are one of the main types of deep learning anomaly detection, which use an encoder-decoder architecture and are trained through a self-supervised reconstruction input pre-task. This method expects that normal samples can be accurately reconstructed, while abnormal samples have high reconstruction errors. Early methods include LSTM-based encoder-decoder models and other models based on random recurrent neural networks extended from LSTM-VAE, such as LSTM-VAE, OmniAnomaly, and InterFusion. Another branch of reconstruction-based methods is deep generative models. MAD-GAN and BeatGAN are variants of generative adversarial networks customized for time series anomaly detection. Anomaly Transformer introduces anomaly attention mechanisms to simultaneously model priors and sequence associations. However, existing reconstruction-based methods may suffer from the problem of over-generalization. This may happen if the encoder extracts unique features of the anomaly or the decoder has excessive decoding power for the anomaly encoding vector, which in turn leads to problems such as missed, low precision, and long time for time series anomaly detection. Summary of the invention
[0005] In view of the shortcomings of the existing technology, the present invention proposes a multivariate time series anomaly detection method based on abnormal attention and gated storage. Taking into account the characteristics of multivariate time series, the reconstruction process is improved on the basis of abnormal attention, thereby improving the model generalization ability and detection accuracy.
[0006] A multivariate time series anomaly detection method based on abnormal attention and gated storage is used to detect multivariate time series anomaly detection model based on abnormal attention and gated storage. The specific steps are as follows: Step 1: Collect a multivariate time series dataset, preprocess the time series in the feature dimension, and then divide it into a training set and a test set; Step 2: Build a multivariate time series anomaly detection model based on abnormal attention and gated storage, including: Abnormal attention module, gated memory module and weak decoder module; Step 3: Input the preprocessed time series into the abnormal attention module to obtain the prior association matrix of the time series data and the sequence association matrix The correlation difference is based on the sequence correlation matrix Get the hidden state data of time series data features ; Step 4: Hidden state data As a query Input gated storage module, continuously updating query and memory items , get the retrieved memory item ,Will and The input data of the weak decoder is obtained by adding the feature dimensions, and the input data is reconstructed by the weak decoder to obtain a reconstructed time series; Step 5: Define the loss function and hyperparameters based on reconstructed time series and associated differences, and train the multivariate time series anomaly detection model based on abnormal attention and gated storage in two stages. In the first stage, query is generated by associated differences. , for the query Clustering to obtain memory items The initial value of , the second stage trains the gated storage module to iterate continuously; Step 6: Input the test set into the trained model, define the thresholds for normal data and abnormal data, apply the detection criteria based on two-dimensional deviation, determine the abnormal data by comparing the thresholds and anomaly scores, and implement multivariate time series anomaly detection.
[0007] Furthermore, the step 3 specifically includes the following steps: S31. Use the learnable Gaussian kernel to calculate the prior association matrix of the lth layer : P l = Rescale( [ 1 2π σ i exp(− |j−i| 2 2 σ i 2 )] i,jϵ{1,...,N} ) (1) In the formula, is the Gaussian kernel parameter at the ith time point, To represent an exponential function with a natural number e as base, we use The calculated association weights are converted into discrete distributions; the Gaussian kernel parameter σ is calculated as follows: (2) In the formula, Indicates The output of the layer, It is Gaussian kernel parameter weight matrix of layer abnormal attention; S32, use weight matrices Q and K to calculate the sequence association matrix of the lth layer : (3) In the formula, is the query matrix, K is the key matrix, represents the dimension of the model, represents locality sensitive hashing attention, Standardize the sequence weights. Q and K are calculated as follows:
[0008] (4) In the formula, Indicates The output of the layer, , They are The query weight matrix and key weight matrix of layer anomaly attention; LSHA( ) Calculate the attention of different local blocks according to the hash value. When the local sensitive hashing method is used to calculate When and In the same local block, locality sensitive hashing attention The calculation method is as follows: (5) In the formula, represents the query vector corresponding to each input sequence element i, Represents the key vector corresponding to each input sequence element j, and L is the length of the input sequence; S33, calculate the first Hidden representation of layers : (6) In the formula, is the value matrix, For the The output of the layer, Representation layer normalization; where, The calculation method is as follows: (7) In the formula, Indicates The output of the layer, It is The value weight matrix of layer abnormal attention; S34, by The hidden representation of the layer is calculated Output of the layer , recorded as hidden state data: (8) In the formula, represents a 1D convolution operation, Representation layer normalization.
[0009] Further, in step 4, the memory item of the gated storage module contains typical normal mode data of the input time series, and the updated memory item contains all query vectors corresponding to normal timestamps The prototype defines the memory attention weight under the query condition as follows: (9) In the formula, Represents the i-th memory item, where i is an integer from 1 to the total number of memory items M. represents the query vector of subsequence s at time t, and L is the length of the subsequence; The memory update mechanism calculated according to the memory attention weight is as follows: (10) In the formula, is the L2 norm; The query of the gated storage module comes from the hidden state data in step 2, similar to the memory attention weight under the query condition. The query attention weight under the memory condition is defined as follows: (11) The retrieved memory items after evaluating the query are as follows: (12) Will retrieve the memory item and query Add in the feature dimension to get the input data of the weak decoder; The weak decoder is composed of two fully connected layers, and a reconstructed sequence is obtained after decoding.
[0010] Furthermore, in step 5, a K-mean clustering method is used to obtain an initial value of the memory.
[0011] A second object of the present invention is to provide a multivariate time series anomaly detection system based on abnormal attention and gated storage for implementing the above method, comprising: Data acquisition module, used to obtain multivariate time series data sets; Data preprocessing module, used to preprocess multivariate time series data sets; The multivariate time series anomaly detection module uses the trained, verified and tested multivariate time series anomaly detection model based on abnormal attention and gated storage to detect anomalies in multivariate time series datasets.
[0012] A third object of the present invention is to provide an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above method.
[0013] The fourth object of the present invention is to provide a machine-readable storage medium, characterized in that the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above method.
[0014] The present invention has the following beneficial effects: 1. The present invention applies the gated storage module to the time series reconstruction model based on abnormal attention, and improves the update module and retrieval module therein, thereby enhancing the reconstruction capability of the model, and being able to more effectively capture associated differences, thereby improving the accuracy of time series anomaly detection. It can be widely used in time series anomaly detection of sensor data such as water treatment equipment, traffic signal equipment, and Internet company servers.
[0015] 2. The present invention defines a new method for calculating anomaly scores based on the detection criterion of two-dimensional deviation. The loss function that combines reconstruction loss, association difference loss and entropy loss enhances the robustness and stability of the model and improves the detection accuracy on multiple multivariate time series data sets.
[0016] 3. The present invention introduces local sensitive hashing into the abnormal attention mechanism, which reduces the memory occupied by the algorithm and speeds up the detection of timing anomalies. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 Schematic diagram of a multivariate temporal anomaly detection model based on abnormal attention and gated storage; Figure 2 This is a schematic diagram of the update module structure; Figure 3This is a schematic diagram of the retrieval module structure. DETAILED DESCRIPTION
[0018] The technical solution of the present invention is further described below.
[0019] like Figure 1 As shown, the present invention discloses a multivariate time series anomaly detection method based on abnormal attention and gated storage, comprising the following steps: Step 1: Collect a multivariate time series dataset, including: Four real-world multivariate time series datasets are collected, and the time series are preprocessed in the feature dimension. First, the time series data is converted into a distribution with a mean of 0 and a standard deviation of 1. Value embedding is achieved through one-dimensional convolution, and position embedding is calculated based on sin and cos functions. The two are combined to obtain the final embedding representation, and then the data is divided into several sliding windows.
[0020] The data set may be sensor data from a critical infrastructure system in continuous operation in a water treatment plant, including sensor values such as water level, flow, etc., as well as operational data on facilities such as valves and pumps; the data set may be sensor data from different servers of a large Internet company over a period of several weeks, with each sensor representing a different performance indicator.
[0021] Step 2: Build a multivariate time series anomaly detection model based on abnormal attention and gated storage. This model uses abnormal attention to obtain data features, and inputs the features into the gated storage module and weak encoder to reconstruct the input data.
[0022] The model building process includes: Step 2.1: Construct an abnormal attention module, including: Using a learnable Gaussian kernel, calculate the prior association matrix of the lth layer : P l = Rescale( [ 1 2π σ i exp(− |j−i| 2 2 σ i 2 )] i,jϵ{1,...,N} ) (1) In the formula, is the Gaussian kernel parameter at the i-th time point , To represent an exponential function with a natural number e as base, we use The calculated association weights are converted into discrete distributions; where the Gaussian kernel parameter The calculation method is as follows: (2) In the formula, Indicates The output of the layer, It is Gaussian kernel parameter weight matrix of layer abnormal attention; Use weight matrices Q and K to calculate the sequence association matrix of the lth layer : (3) In the formula, is the query matrix, K is the key matrix, represents the dimension of the model, represents locality sensitive hashing attention, Standardize the sequence weights. Q and K are calculated as follows:
[0023] (4) In the formula, Indicates The output of the layer, , They are The query weight matrix and key weight matrix of layer anomaly attention; LSHA( ) Calculate the attention of different local blocks according to the hash value. When the local sensitive hashing method is used to calculate When and In the same local block, locality sensitive hashing attention The calculation method is as follows: (5) In the formula, represents the query vector corresponding to each input sequence element i, Represents the key vector corresponding to each input sequence element j, and L is the length of the input sequence; According to the serial correlation calculation Hidden representation of the layer: (6) In the formula, is the value matrix, For the The output of the layer, Representation layer normalization; where, The calculation method is as follows: (7) In the formula, Indicates The output of the layer, It is The value weight matrix of layer abnormal attention; By The hidden representation of the layer is calculated Output of the layer : (8) In the formula, represents a 1D convolution operation, Representation layer normalization.
[0024] Step 2.2: Build a gated storage module, including: The gated storage module includes an update module and a retrieval module; The memory item of the gated storage module contains typical normal pattern data of the input time series, and the updated memory item contains all query vectors corresponding to normal timestamps The prototype, such as Figure 2 As shown, in the update module, an incremental method is adopted to define the memory attention weight under the query condition by the memory item and the query vector in the update module. It is calculated by the cosine similarity (C) and softmax (S) The dot product between each memory item and the query is as follows: (9) In the formula, Represents the i-th memory item, where i is an integer from 1 to the total number of memory items M. represents the query vector of subsequence s at time t, and L is the length of the subsequence; The memory update mechanism calculated according to the memory attention weight is as follows: (10) In the formula, is the L2 norm (W); The query of the gated storage module comes from the hidden state data in step 2, similar to the memory attention weight under the query condition, such as Figure 3 As shown in Figure 2, in the retrieval module, an updated query is generated and then fed as input to the decoder. Similar to the gated memory update stage, the memory-conditioned query attention weight is defined by the query vector and the memory item in the retrieval module. It is calculated by the cosine similarity (C) and the softmax (S) by calculating the dot product between each memory item and the query, as shown below:
[0025] (11) The memory item to be retrieved Weighted sum, take As their corresponding weights, the retrieved memory items after the query are calculated as follows: (12) Will retrieve the memory item and query Add in the feature dimension to get the input data of the weak decoder.
[0026] Step 2.3: Construct a weak decoder, which consists of two fully connected layers.
[0027] Step 3: Define the loss function to prepare for model training, including: Set the loss function to: (13) In the formula, To rebuild the losses, is the association difference loss, is the entropy loss, is the hyperparameter set; Among them, the reconstruction loss is calculated as follows: (14) In the formula, is the reconstructed sequence, represents the L2 norm; The associated discrepancy loss is calculated as follows: (15) In the formula, represents the KL divergence, L is the total number of layers of the abnormal attention module, represents the i-th row of the sequence association matrix of the l-th abnormal attention layer, represents the i-th row of the prior association matrix of the l-th abnormal attention layer; The entropy loss is calculated as follows: (16) in, is the query attention weight under memory condition.
[0028] Step 4: Input the training set of the time series data set into the model. The model is trained in two stages. In the first stage, queries are generated by association differences, and K-mean clustering is performed on the queries to obtain the initial values of the memory items. In the second stage, the update module and retrieval module of the gated storage module are trained to work together to update the parameters.
[0029] Step 5: Infer the anomaly detection model on the test set, define the thresholds of normal data and abnormal data, apply the detection criteria based on two-dimensional deviation, and determine the abnormal data by comparing the thresholds and anomaly scores.
[0030] The anomaly score is defined as follows: A( X s ) = softmax( [LSD( q t s ,m)] t=1,...,L )∘ [ISD( X t,: s , X ̂ t,: s )] t=1,...,L (17) in, is element-by-element multiplication, LSD is the latent space deviation, and ISD is the input space deviation, which are defined as: (18) (19) in, is the query at timestamp t, is the memory item closest to it, is the atomic sequence, is the reconstructed subsequence.
[0031] The threshold is set to the top p% of the combined anomaly scores of the training data and the validation data. The specific value of p and the statistical details of each data set are shown in Table 1: Table 1 Statistical details of each data set Train Vaild Test p(%) Dim SMD 566724 141681 708420 0.5 38 MSL 46653 11664 73729 1.0 55 PSM 105984 26497 87841 1.0 26 SMAP 108146 27037 427617 1.0 25 In order to test the advantages of the proposed method in multivariate time series anomaly detection, this method is compared with THOC, LSTM-VAE, BeatGAN, OmniAnomaly, InterFusion, and Anomaly Transformer. The experimental results are shown in the following table: Table 2 Results of each detection method From the data in Table 2, we can see that the final F1-score of this method is higher than that of other methods.
[0032] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be considered exemplary and non-restrictive in all respects, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.
Claims
1. A multivariate time series anomaly detection method based on abnormal attention and gated storage, characterized in that: The following steps are involved: Step 1: Collect a multivariate time series data set, preprocess the multivariate time series in the feature dimension, and then divide it into a training set and a test set; Step 2: Build a multivariate time series anomaly detection model based on abnormal attention and gated storage, including: Abnormal attention module, gated memory module and weak decoder module; Step 3: Input the preprocessed time series into the abnormal attention module to obtain the prior association matrix of the time series data and the sequence association matrix The correlation difference is based on the sequence correlation matrix Get the hidden state data of time series data features ; Step 4: Hidden state data As a query Input gated storage module, update query and memory items of gated storage modules , get the retrieved memory item ,Will and The input data of the weak decoder is obtained by adding the feature dimensions, and the input data is reconstructed by the weak decoder to obtain a reconstructed time series; Step 5: Define the loss function and hyperparameters based on reconstructed time series and associated differences, and train the multivariate time series anomaly detection model based on abnormal attention and gated storage in two stages. In the first stage, query is generated by associated differences. , for the query Clustering to obtain memory items The initial value of, the second stage trains the iteration parameters of the gated storage module; Step 6: Input the test set into the trained model, define the thresholds for normal data and abnormal data, apply the detection criteria based on two-dimensional deviation, determine the abnormal data by comparing the thresholds and anomaly scores, and implement multivariate time series anomaly detection.
2. The multivariate time series anomaly detection method based on abnormal attention and gated storage according to claim 1 is characterized in that: The step 3 specifically includes the following steps: S31. Use a learnable Gaussian kernel to calculate the Prior correlation matrix of the layer : (1) In the formula, is the Gaussian kernel parameter at the i-th time point , represents an exponential function with the natural number e as base, The calculated association weights are converted into discrete distributions; where the Gaussian kernel parameter The calculation method is as follows: (2) In the formula, Indicates The output of the layer, It is Gaussian kernel parameter weight matrix of layer abnormal attention; S32, using weight matrices Q, K to calculate Sequential correlation matrix of the layer : (3) In the formula, is the query matrix, K is the key matrix, represents the dimension of the model, represents locality sensitive hashing attention, Indicates that the sequence weights are standardized; Q and K are calculated as follows: (4) In the formula, Indicates The output of the layer, Indicates The query weight matrix of layer anomaly attention, Indicates Key weight matrix for layer outlier attention; Locality Sensitive Hashing Attention The calculation method is as follows: (5) In the formula, represents the query vector corresponding to each input sequence element i, Represents the key vector corresponding to each input sequence element j, and L is the length of the input sequence; S33, according to the sequence association matrix Calculate the Hidden representation of layers : (6) In the formula, is the value matrix, For the The output of the layer, Representation layer normalization; where, The calculation method is as follows: (7) In the formula, represents the output of the l-1th layer, is the value weight matrix of the abnormal attention at the lth layer; S34, by Hidden representation of layers Calculate the output of the lth layer, recorded as hidden state data : (8) In the formula, represents a 1D convolution operation, Representation layer normalization.
3. The multivariate time series anomaly detection method based on abnormal attention and gated storage according to claim 1 is characterized in that: In step 4, the gated storage module includes an update module and a retrieval module; In the update module, the hidden state data As a query vector , the memory attention weight under the query condition is defined as follows: (9) In the formula, Represents the i-th memory item, where i is an integer from 1 to the total number of memory items M. represents the query vector of subsequence s at time t, and L is the length of the subsequence; The memory update mechanism calculated according to the memory attention weight is as follows: (10) In the formula, is the L2 norm.
4. The multivariate time series anomaly detection method based on abnormal attention and gated storage according to claim 3 is characterized in that: In the retrieval module, the query attention weight under memory condition is defined as follows: (11) The retrieved memory items after evaluating the query are as follows: (12) Will retrieve the memory item and query Add in the feature dimension to get the input data of the weak decoder.
5. The multivariate time series anomaly detection method based on abnormal attention and gated storage according to claim 1 is characterized in that: The weak decoder consists of two fully connected layers.
6. The multivariate time series anomaly detection method based on abnormal attention and gated storage according to claim 1 is characterized in that: In step 5, set the loss function to: (13) In the formula, To rebuild the losses, is the association difference loss, is the entropy loss, is the hyperparameter set; Among them, the reconstruction loss is calculated as follows: (14) In the formula, is the reconstructed sequence, represents the L2 norm; The associated discrepancy loss is calculated as follows: (15) In the formula, represents the KL divergence, L is the total number of layers of the abnormal attention module, Represents the l-th level sequence association matrix The i-th row of Represents the l-th layer prior association matrix The i-th row of The entropy loss is calculated as follows: (16) in, is the query attention weight under memory condition.
7. The multivariate time series anomaly detection method based on abnormal attention and gated storage according to claim 1 is characterized in that: In step 6, the anomaly score is defined as: (17) in, is element-by-element multiplication, LSD is the latent space deviation, and ISD is the input space deviation, which are defined as: (18) (19) in, is the query vector of subsequence s at time t, For Recent memory items, is the atomic sequence, is the reconstructed subsequence.
8. A multivariate time series anomaly detection system based on abnormal attention and gated storage that implements the method of any one of claims 1 to 7, characterized in that include: Data acquisition module, used to obtain multivariate time series data sets; Data preprocessing module, used to preprocess multivariate time series data sets; The multivariate time series anomaly detection module uses the trained, verified and tested multivariate time series anomaly detection model based on abnormal attention and gated storage to detect anomalies in multivariate time series datasets.
9. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the method according to any one of claims 1 to 7.
10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the method described in any one of claims 1 to 7.
Citation Information
Patent Citations
Multivariable time sequence anomaly detection method based on hierarchical attention network
CN116361640A
Time series data anomaly detection method and system and computer readable medium
CN117633054A
Cited By
Time series data anomaly detection method and system based on flattening attention mechanism
CN119336821A
VAE and association difference-based multivariate time series data anomaly detection method
CN119783010A
Multi-dimensional time series data anomaly discrimination and restoration method based on convolutional neural network
CN119961847A
Factory data anomaly detection method and system based on big data analysis
CN120744783A
Equipment system anomaly detection method and system based on dual-path adaptive learning
CN121387618A
