A time series data anomaly detection method based on multi-head attention model

Abstract

The present invention relates to a method for detecting anomalies in time series data based on a multi-head attention model, and belongs to the technical field of anomaly detection in computer databases. The present invention builds a network framework for detecting anomalies in time series data based on a multi-head attention model, which can process multi-source time series data, has robustness and good generalization ability, and can achieve faster network training and more accurate abnormal sample detection. A frequency domain analysis module is introduced to make full use of the frequency domain information of time series data. By performing frequency domain analysis on the input sequence and the output sequence, the model can obtain effective sequence information. By measuring the distance between the two frequency domain analysis results, the network can be supervised to generate a more accurate output sequence, thereby obtaining a more accurate abnormal judgment threshold and achieving more accurate anomaly detection. The present invention has obvious advantages in the feature extraction capability of time series data and the detection accuracy of abnormal samples.

Description

Technical Field

[0001] The present invention relates to a time series data anomaly detection method based on a multi-head attention model, and belongs to the technical field of computer database anomaly detection. Background Art

[0002] In recent years, the Internet of Things (IoT) has been widely studied and applied. By seamlessly deploying a large number of sensors and controllers, the entire production cycle of a product can be simulated, predicted, and controlled. Sensors collect data at a certain frequency and send it to the corresponding data receiver. The data received by the receiver is time series data, which is strictly sorted and stored in chronological order. The contextual data points of time series data are highly causally related and have significant time dependence. Due to the large scale of data and the complexity of calculation and storage, data anomalies are inevitable, and these anomalies may pose a great risk to the security of the entire IoT system. Therefore, effective monitoring and detection of anomalies in time series data is crucial to discover faults and avoid potential risks.

[0003] Anomalies refer to samples that are significantly different from most other samples in terms of features. The anomaly detection task requires identifying a small number of abnormal samples from a large number of normal samples. At present, the data frame rate collected by devices in large IoT systems is very high each time, and with the upgrade of sensor technology and the improvement of data transmission capacity, the length of data time series is also increasing. In addition, time series data lacks labels and has diverse anomalies. However, traditional machine learning algorithms cannot model this high-dimensional data well.

[0004] Deep learning models are good at modeling high-dimensional data from multiple sources because of their strong data expression capabilities. They can also extract effective contextual information from time series data and have good generalization capabilities. Therefore, unsupervised deep learning algorithms can be used to detect anomalies in time series data.

[0005] However, traditional models have limited ability to obtain contextual information for dense time series data, and their computational speed is slow and costly, making them unable to simulate the long-term trend of data well. Moreover, most methods focus on modeling in the time domain, but ignore the frequency domain information of time series data. Summary of the invention

[0006] The purpose of the present invention is to creatively propose a method for detecting anomalies in time series data based on a multi-head attention model in view of the shortcomings and improvement needs of the prior art. This method adopts advanced technologies such as multi-head attention and combined analysis of time domain and frequency domain, which effectively improves the model's ability to extract sample feature information and the accuracy of detecting a variety of abnormal samples from a large number of correct samples, effectively solves the problem of anomaly detection in time series data, makes anomaly detection in time series data more efficient and accurate, and provides better detection results for the monitoring system.

[0007] The present invention is implemented by adopting the following technical solutions.

[0008] A method for detecting anomalies in time series data based on a multi-head attention model comprises the following steps:

[0009] Step 1: Data preprocessing.

[0010] Since the attribute value ranges of data collected by raw IoT sensors in different situations vary greatly, the model will be insensitive to changes in smaller values. Therefore, the raw data needs to be normalized across the entire time dimension based on contextual data in the same data source.

[0011] During the preprocessing process, the time series data points are normalized according to the maximum and minimum values ​​of their columns.

[0012] Step 2: Build the encoder.

[0013] In order to extract features containing effective contextual information, the present invention constructs an encoder based on a multi-head attention model.

[0014] The normalized time series obtained in step 1 is encoded for subsequent network information transmission. The sequence obtains the potential expression of the feature through masked multi-head attention, feedforward network, and multi-head attention structure, which enables the model to focus on the target area that needs to be focused on, and at the same time combines the global information of the input data.

[0015] Furthermore, the encoder is composed of various multi-head attention modules and feed-forward networks, which are used to expand the information processing scope of the model, filter out effective information, integrate contextual information in the sequence, and obtain representative latent representations.

[0016] Step 3: Construct the memory module.

[0017] In order to reconstruct a better feature expression, the present invention constructs a memory module, which takes the encoded potential representation as input, initializes a memory item, and then measures the distance between the potential representation and the memory item. If the distance is less than the set threshold, the potential representation is identified as a normal sample and should participate in the reconstruction of the feature expression operation with a greater weight. The memory item should be as sparse as possible, and complex combination structures should be avoided as much as possible to better reconstruct the expression, and the memory item should remember the typical normal mode to make the reconstruction more reliable.

[0018] Furthermore, the memory module uses the distance between the latent representation and the memory items to assign weights to the features involved in the reconstruction and trains memory items with typical normal patterns.

[0019] Step 4: Build the decoder.

[0020] The decoder corresponding to the encoder is used to restore the reconstructed features to a sequence expression. Specifically, the reconstructed feature expression can be restored to a sequence data structure through a feedforward network.

[0021] Step 5: Frequency domain analysis.

[0022] In order to obtain richer sequence information, both the normalized input sequence and the decoded output sequence are subjected to frequency domain analysis. First, the time domain sequence is converted into a frequency domain sequence through discrete Fourier transform, and then the frequency domain features are extracted through a time convolutional network. Finally, the frequency domain analysis results of the input sequence and the output sequence are measured. The network training auxiliary model is used to obtain the optimal parameters and reconstruction features.

[0023] Step 6: Anomaly identification.

[0024] In order to obtain a suitable threshold to judge whether a sample is abnormal, three loss functions are used to train the network, including reconstruction error loss function, sparse penalty loss function and frequency domain analysis loss function, to obtain suitable network parameters and output sequence, and then calculate the threshold for abnormal judgment.

[0025] Among them, the reconstruction error loss function makes the reconstructed feature expression more accurate, the sparse penalty loss function makes the memory item assign greater weight to normal samples, and the frequency domain analysis loss function makes the anomaly score more accurate.

[0026] Beneficial Effects

[0027] Compared with the prior art, the present invention has the following advantages:

[0028] 1. A time series data anomaly detection model based on a multi-head attention module was built. Compared with the prior art, the present invention builds a network framework for time series data anomaly detection based on a multi-head attention model, which can process multi-source time series data, has robustness and good generalization ability, and can achieve faster network training and more accurate abnormal sample detection.

[0029] 2. A more reliable encoding and decoding strategy is designed. The present invention designs an encoding and decoding strategy based on a multi-head attention mechanism. The multi-head attention mechanism can pay more attention to key information and give greater weight to important information. The present invention combines the masked multi-head attention mechanism, the feedforward network and the multi-head attention mechanism, and has a more powerful feature extraction and representation capability, which can make full use of the rich contextual information of the input sequence to obtain a potential feature expression with more prominent effective information.

[0030] 3. Introduce frequency domain analysis. The present invention introduces a frequency domain analysis module to make full use of the frequency domain information of time series data. By performing frequency domain analysis on the input sequence and the output sequence, the model can obtain effective sequence information. By measuring the distance between the two frequency domain analysis results, the network can be supervised to generate a more accurate output sequence, thereby obtaining a more accurate anomaly judgment threshold and achieving more accurate anomaly detection.

[0031] In summary, through the implementation of the above innovations, the present invention has obvious advantages in the feature extraction capability of time series data and the detection accuracy of abnormal samples. In the case of large input, this method can not only keep the network stable and robust, but also achieve fast reasoning and accurate detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is the overall flow chart of the method of the present invention;

[0033] Figure 2 It is a schematic diagram of the encoder module of the present invention;

[0034] Figure 3 It is a decoder module framework diagram of the present invention;

[0035] Figure 4 This is the framework diagram of the time series data frequency domain analysis module. DETAILED DESCRIPTION

[0036] The present invention is further described in detail below with reference to the accompanying drawings.

[0037] like Figure 1 A method for detecting anomalies in time series data based on a multi-head attention model includes the following steps:

[0038] Step 1: Data preprocessing.

[0039] Get complete time series data from multiple sources from sensors , , Represents feature dimension; Represents a time series data sample collected from a data source. The sample length of each sample is , .

[0040] For each data sample point , use the maximum and minimum values ​​of its column as the normalization parameters for the sample point. Normalized data It is expressed as:

[0041] ,

[0042] in, Indicated in Found The minimum value in the column. Indicated in Found The maximum value in the column.

[0043] Step 2: Build the encoder.

[0044] The encoder uses a multi-head attention structure, which can handle natural language tasks well. Using this encoder to encode the entire input sequence allows the model to jointly pay attention to information from different representation subspaces at different positions, and obtain feature information with contextual connections. Figure 2 shown.

[0045] The multi-head attention model is defined as follows: For a multivariate sequence, it is first converted into a modality Define three matrices Q (query), K (key), and V (value), and the scalar dot product attention model is expressed as:

[0046] ,

[0047] in, represents transpose, is the number of heads of the multi-head attention model. The combined weight of the matrix is ​​obtained through softmax, and this weight is used to compress the matrix V into a smaller embedded expression to simplify reasoning in subsequent neural network operations.

[0048] The attention mechanism weights The term is scaled to reduce the variance of the weights and promote stable training. For the input matrices Q, K, and V, pass them to feed-forward layers, we get , ; The multi-head attention model is expressed as:

[0049] ,

[0050] ,

[0051] in, represents the matrix channel addition operation, represents a single-head attention mechanism, Represents the output features of each single-head attention mechanism.

[0052] First, apply position encoding to obtain the encoder input .

[0053] The encoder then does the following:

[0054] ,

[0055] in, Represents the input matrix Perform multi-head self-attention operations; Represents a matrix addition operation. The operation masks part of the input to hide the window sequence of future timestamps in the same input until the t When the complete sequence of timestamps is fed into the model, the model takes advantage of greater contextual information. represents normalization processing, Represents the output feature representation of the encoder module.

[0056] Using a feed-forward network The multi-head attention module generates attention weights to capture the temporal trend in the input sequence, enabling the model to integrate effective information in the features and use global contextual information for subsequent reasoning and judgment.

[0057] Step 3: Construct the memory module.

[0058] The encoder generates a latent representation of the temporal data. , define a memory module , which is caused by N Memory items , For the The addressing vector is obtained by calculating the distance between the potential feature representation and each memory item. :

[0059] ,

[0060] in, Indicates similarity, Indicates transpose.

[0061] The inner product is used to calculate the matching degree between the potential representation and each memory item, and then compared with other memory items through softmax to obtain the relative similarity distance; the similarity distance of each memory item finally constitutes the addressing vector.

[0062] Similarity Memory items above the set threshold are more representative of the normal mode of the sample and will participate in reconstructing the potential representation. :

[0063] ,

[0064] in, represents the memory item matrix, Indicates the number of memory items.

[0065] Step 4: Build the decoder.

[0066] like Figure 3 As shown. The decoder uses a feedforward network structure to decode the features, which is expressed as:

[0067] ,

[0068] in, is the activation function, outputting values ​​in the range [0,1]; represents the output of the decoder, that is, the reconstructed sequence, , , Indicates A sample of data sources, .

[0069] Step 5: Frequency domain analysis.

[0070] By performing frequency domain analysis on time series data, we can obtain rich information about the time series, such as Figure 4 shown.

[0071] The data in the time series is usually discrete, so the discrete Fourier transform is first used to transform the time series data from the time domain to the frequency domain. , whose discrete Fourier transform It is expressed as:

[0072] ,

[0073] in, is the circumference of a circle, Indicates scientific notation; represents the number of time series samples, .

[0074] The result of the discrete Fourier transform contains the real part and the imaginary part ;Will and Intersect, get Through discrete Fourier transform, the normalized input sequence is obtained , Frequency domain expression of , and the decoder outputs the reconstructed sequence , Frequency domain expression of .

[0075] The obtained frequency domain expressions are input into the temporal convolutional network TCN to learn the feature representation in the frequency domain. The TCN structures in the two TTFs are the same but the parameters are not shared. After that, the analysis results of the data in the frequency domain are obtained through maximum pooling, dimensionality reduction, linear layer and layer normalization operations.

[0076] By comparing the similarity of the frequency domain analysis results of the input sequence and the output sequence, the abnormality of the data sample can be determined.

[0077] Step 6: Anomaly identification.

[0078] During the training process, normal samples and abnormal samples are trained together without labeling. This model extracts typical features in the memory items by supervising the effect of the reconstructed samples of the output, so the reconstruction error needs to be minimized. , measured using mean square error :

[0079] ,

[0080] in, is the learnable parameter of the memory, is the input time series data sample, To reconstruct time series data samples, is the number of data in the sample, Represents 2-norm calculation.

[0081] Addressing Vector It should be as sparse as possible, not only to avoid complex combinations so as to better reconstruct abnormal samples, but also to make each memory item remember the typical normal mode. Therefore, the present invention designs a sparse penalty loss function , making the sparse addressing vector sparse:

[0082] ,

[0083] Frequency domain analysis results of input and output sequences are used to measure the degree of abnormality :

[0084] ,

[0085] in, and Respectively represent the frequency domain analysis results of the input sequence and the output sequence, the distance function Use cosine similarity.

[0086] The final loss function for:

[0087] ,

[0088] in, and is an adjustable hyperparameter.

[0089] Through network training, the model learns the optimal parameters and outputs a suitable reconstructed sequence; the distance between the normalized input sequence and the reconstructed output sequence is used as the anomaly score :

[0090] ,

[0091] This anomaly score As the decision point threshold, to determine the degree of deviation of the observed sample, any sample with a deviation value exceeding the threshold is judged as abnormal.

Claims

1. A time series data anomaly detection method based on a multi-head attention model, characterized in that: The following steps are involved: Step 1: Data preprocessing; Obtain complete time series data from multiple sources from IoT sensors. The complete time series data is the original data. The raw data is normalized across the entire time dimension based on contextual data from the same data source; During the preprocessing process, the time series data points are normalized according to the maximum and minimum values ​​of their columns; Step 2: Build the encoder; Construct an encoder based on a multi-head attention model to encode the normalized time series obtained in step 1. The sequence obtains the potential expression of the feature through masked multi-head attention, feedforward network, and multi-head attention structure, so that the model focuses on the target area that needs to be focused on, and at the same time combines the global information of the input data; The encoder consists of a variety of multi-head attention modules and feedforward networks, which are used to expand the information processing scope of the model, filter out effective information, integrate contextual information in the sequence, and obtain representative potential representations; Step 3: Construct memory module; This module takes the encoded latent representation as input, initializes a memory item, and then measures the distance between the latent representation and the memory item. If the distance is less than the set threshold, the latent representation is considered a normal sample and will participate in the reconstruction of feature expression with a greater weight. The memory module uses the distance between the potential representation and the memory item to assign weights to the features involved in the reconstruction and trains the memory items with typical normal patterns. Step 4: Build the decoder; The decoder corresponding to the encoder is used to restore the reconstructed features to sequence expressions; Step 5: Frequency domain analysis; Perform frequency domain analysis on both the normalized input sequence and the decoded output sequence; first, convert the time domain sequence into a frequency domain sequence through discrete Fourier transform, then extract the frequency domain features through a time convolutional network, and finally measure the frequency domain analysis results of the input sequence and the output sequence; Through network training auxiliary model, the optimal parameters and reconstruction features are obtained; Step 6: Anomaly identification; Three loss functions are used to train the network, including reconstruction error loss function, sparse penalty loss function and frequency domain analysis loss function, to obtain network parameters and output sequence, and then calculate the threshold of abnormal judgment; Among them, the reconstruction error loss function makes the reconstructed feature expression more accurate, the sparse penalty loss function makes the memory item assign greater weight to normal samples, and the frequency domain analysis loss function makes the anomaly score more accurate.

2. The method for detecting anomalies in time series data based on a multi-head attention model according to claim 1, characterized in that: In step 1, complete time series data from multiple sources is obtained from sensors Represents feature dimension; X i Represents a time series data sample collected from a data source, and the sample length of each sample is T. For each data sample point Use the maximum and minimum values ​​of the column as the normalization parameters of the sample point; normalize the data It is expressed as: in, Indicates that in X The minimum value in the column. Indicates that in X The maximum value in the column.

3. The method for detecting anomalies in time series data based on a multi-head attention model according to claim 1, characterized in that: In step 2, for a multivariate sequence, first convert it into a matrix with mode h, define three matrices Q: query, K: key, V: value, and the scalar dot product attention model is expressed as: Where T represents transpose, h is the number of heads in the multi-head attention model; the combined weight of the matrix is ​​obtained through softmax, and this weight is used to compress the matrix V into a smaller embedded expression; The scalar dot product attention model divides the weights into The term is scaled; for the input matrix Q, K, V, it is passed to h feed-forward layers to obtain Q i ,K i ,V i , i∈{1,2…,h}; the multi-head attention model is expressed as: MultiHeadAtt(Q,K,V)=concat(H1,H2,…,H h ) H i =That(Q i ,K i ,V i ) Among them, concat represents matrix channel addition, Att represents a single-head attention mechanism, and H i Represents the output features of each single-head attention mechanism; First, position encoding is applied to obtain the encoder input I0; The encoder then does the following: I1=Mask(MultiHeaddAtt(I0, I0, I0)) I2=LayerNorm(I0+I1) I3=LayerNorm(I2+FeedForward(I2)) F=LayerNorm(I3+MultiHeadAtt(I3, I3, I3)) Among them, MultiHeadAtt(I0,I0,I0) represents the multi-head self-attention operation on the input matrix I0; + represents the matrix addition operation; Mask(·) operation masks part of the input to hide the window sequence of future timestamps in the same input until the complete sequence of the t-th timestamp is input to the model, and the model uses greater context information; LayerNorm represents the normalization process, and F represents the output feature expression of the encoder module; The feedforward network FeedForward and the multi-head attention module are used to generate attention weights and capture the temporal trend in the input sequence, so that the model can integrate effective information in the features and use global context information for subsequent reasoning and judgment.

4. The method for detecting anomalies in time series data based on a multi-head attention model according to claim 1, characterized in that: In step 3, the encoder generates a latent representation of the time series data Define a memory item matrix It consists of N memory items Composition, e i is the i-th memory item; the addressing vector is obtained by calculating the distance between the potential feature representation and each memory item z i =Softmax(F T e i ) Among them, z i represents similarity, T represents transposition; The inner product is used to calculate the matching degree between the potential representation and each memory item, and then compared with other memory items through softmax to obtain the relative similarity distance; the relative similarity distance of each memory item finally constitutes the addressing vector; Similarity z i Memory items above the set threshold are more representative of the normal mode of the sample, and the memory items are involved in reconstructing the potential representation. Where E represents the memory item matrix, and N represents the number of memory items; In step 4, the decoder uses a feedforward network structure to decode the features, expressed as: Among them, Sigmoid is the activation function, which outputs a value in the range of [0,1]; O represents the output of the decoder, that is, the reconstructed sequence, O i represents the i-th data source sample, 5. The method for detecting anomalies in time series data based on a multi-head attention model according to claim 1, characterized in that: In step 5, firstly, discrete Fourier transform is used to transform the time series data from the time domain to the frequency domain; For a time series sample x a ={x0,x1,…,x A-1 }, the result after discrete Fourier transform is X k ={X0,X1,…,X A-1 }, X k It is expressed as: Where π is the circumference of a circle, e represents scientific notation, A represents the number of time series samples, 0≤k≤A-1; The result of discrete Fourier transform contains real part Re and imaginary part Im; intersecting Re and Im, we get X k ={Re1,Im1,Re2,Im2,…,Re n ,Im n }; Through discrete Fourier transform, the normalized input sequence X is obtained i The frequency domain expression of X ki , and the decoder outputs the reconstructed sequence O i Frequency domain expression of O ki , i∈{1,2…,T}; The obtained frequency domain expressions are input into the temporal convolutional network (TCN) to learn the feature representation in the frequency domain. The TCN structures in the two TTFs are the same but the parameters are not shared. After that, the analysis results of the data in the frequency domain are obtained through the maximum pooling, dimensionality reduction, linear layer and normalization layer operations. By comparing the similarity of the frequency domain analysis results of the input sequence and the output sequence, the abnormality of the data sample can be determined.

6. The method for detecting anomalies in time series data based on a multi-head attention model according to claim 1, characterized in that: In step 6, minimize the reconstruction error L rec , using mean square error to measure L rec : Among them, θ E is the learnable parameter of the memory, X is the input time series data sample, O is the reconstructed time series data sample, n is the number of data in the sample, and ‖·‖2 represents the 2-norm calculation; Among them, the addressing vector Z should be as sparse as possible, and the sparse penalty loss function L is set sparse , making the sparse addressing vector sparse: The frequency domain analysis results of the input sequence and the output sequence are used to measure the degree of abnormality L TTF : <h2 style=";text-align:left;direction:ltr">L<h2 style=";text-align:left;direction:ltr"> TTF <h2 style=";text-align:left;direction:ltr"> =dis(TTF<h2 style=";text-align:left;direction:ltr"> I <h2 style=";text-align:left;direction:ltr"> TTF<h2 style=";text-align:left;direction:ltr"> o <h2 style=";text-align:left;direction:ltr"> ) Among them, TTF I and TTF o Represent the frequency domain analysis results of the input sequence and the output sequence respectively, and the distance function dis() uses cosine similarity; The final loss function L is: L=L rec +αL sparse +βL TTF Among them, α and β are adjustable hyperparameters; Through network training, the model learns the optimal parameters and outputs the reconstructed sequence; the distance between the normalized input sequence and the reconstructed output sequence is used as the anomaly score δ: δ=‖XO‖2 This anomaly score δ is used as the decision point threshold to determine the degree of deviation of the observed sample. Any sample with a deviation value exceeding the threshold is judged as an anomaly.

Images