Time sequence anomaly detection method based on multi-model fusion

Abstract

The invention discloses a time sequence anomaly detection method based on multi-model fusion, and is suitable for the technical field of industrial data anomaly detection. According to the method, an anomaly detection model named DDI-Net is provided for the complex characteristics of a multivariable time sequence in an industrial scene, the advantages of a de-noising diffusion model and an Informer model are combined, and the method specifically comprises the following steps that the time complexity and the memory use efficiency of O (LlogL) are achieved through a ProbSparse self-attention mechanism, and the calculation efficiency is remarkably improved; preferentially processing the dominant attention score in the N-layer stacked structure by utilizing a distillation self-attention mechanism, and sharply reducing the total space complexity to O ((2-belongs to) LlogL); and an abnormal score is calculated by comparing the difference between the generated sequence and the original sequence, so that accurate detection is realized. According to the method, the F1 score is up to 98.26% on five industrial data sets such as SMD and SWaT, the training efficiency is improved by more than 30% compared with similar methods, meanwhile, memory consumption is reduced, the training speed is increased, and the method is suitable for industrial system real-time monitoring and high-dimensional data rapid analysis scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial data anomaly detection, and particularly to a time series anomaly detection method based on multi-model fusion. Background Art

[0002] Under the background of Industry 4.0, with the development of industrial Internet and intelligent manufacturing technologies, and the progress of sensing and monitoring technologies, the operating states of modern devices can be recorded in real time, thus generating and accumulating a large amount of industrial multivariate time series data; time series analysis involves modeling these data to achieve classification, prediction, and anomaly detection; among them, anomaly detection is an important research direction in time series analysis; time series anomaly detection refers to detecting outliers or abnormal patterns in time series data through analysis and modeling; time series anomaly detection has been widely studied in multiple application fields, especially in fault diagnosis in industrial environments; effective anomaly detection can provide timely warning information, enabling relevant personnel to gain valuable time and prevent disasters from occurring; therefore, a timely and efficient anomaly detection method is crucial.

[0003] In order to achieve effective fault diagnosis in industrial environments, a large number of researchers have started to study industrial multivariate time series data anomaly detection schemes; currently, the more common schemes are to use variational autoencoders (VAEs) and generative adversarial networks (GANs), but VAEs have insufficient accuracy in dealing with discrete feature data due to the assumption of latent space continuity, and the generated samples are not as realistic as those of GANs, while GANs are prone to problems such as gradient disappearance and mode collapse during the training process, and these models are prone to fitting the abnormal parts of time series, affecting the anomaly detection performance; denoising diffusion models, as advanced generative models, have better mode coverage and sample quality, and are more stable in training, but in time series applications, their standard architecture U-Net is difficult to effectively process the deep local features and complex time dependencies of time series; although the Transformer model has made progress in multiple fields, can capture the dependencies between data elements, and can perform parallel computing, it has problems such as quadratic time complexity, high memory usage, and architecture limitations; in view of the many problems existing in the existing methods, designing an anomaly detection method that can effectively capture and identify industrial time series anomalies while greatly saving computing power resources is an urgent problem to be solved by researchers in this field. Summary of the Invention

[0004] Based on the deficiencies of the existing methods, the present invention discloses a time series anomaly detection method based on multi-model fusion, which is used to efficiently capture and identify anomalies in industrial time series data while greatly saving computing power resources.

[0005] To achieve the above object, a time series anomaly detection method based on multi-model fusion disclosed by the present invention includes:

[0006] Perform preliminary reconstruction through a simple autoencoder to obtain a reconstructed time series, and then calculate the error between the reconstructed time series and the original time series to generate a mask matrix;

[0007] A method combining a denoising diffusion model and an Informer model is designed. In this method, the denoising diffusion model adds noise in a Markov chain manner during the forward process and learns how to map back from the latent space to the signal space to remove this noise. Combining the Informer as the neural network structure inside the denoising diffusion model enhances the processing ability of long sequence data and accelerates the model convergence speed at the same time;

[0008] According to the difference between the reconstructed time series generated by the above combined model and the original time series, assign an anomaly score to each node in the time series, and then comprehensively evaluate the anomaly status of the time series of each node.

[0009] According to a time series anomaly detection method based on multi-model fusion provided by the present invention, the working process of the mask mechanism is as follows:

[0010] Use an autoencoder to perform preliminary reconstruction on the time series;

[0011] Calculate the reconstruction error: Each node is associated with a reconstruction error. According to the reconstruction error generated by the autoencoder and the actual anomaly ratio, determine which nodes need to be masked;

[0012] Calculate the multivariate similarity metric: Use the multivariate similarity metric between the masked nodes and their neighboring nodes to determine the size of the mask matrix. If the multivariate similarity metric is greater than the threshold, the mask window expands in both directions, otherwise the expansion stops;

[0013] Generate a mask matrix: According to the reconstruction error and the required size of the mask, generate a mask matrix M, which determines which nodes should be masked during the subsequent self-attention calculation process.

[0014] According to a time series anomaly detection method based on multi-model fusion provided by the present invention, the combination of the denoising diffusion model and the Informer model includes:

[0015] Train the denoising diffusion model and the Informer model to recover the original data from the noisy data, generate high-quality data samples, and capture the complex structure and dynamic changes of the data;

[0016] Replace the U-Net in the denoising diffusion model with the Informer model to better process the deep local features and long-term dependencies in the time series data;

[0017] The convolutional layer and pooling layer of the U-Net in the denoising diffusion model are replaced by the ProbSpare self-attention mechanism and the distillation self-attention mechanism of Informer;

[0018] According to the method for time series anomaly detection based on multi-model fusion provided by the present invention, an anomaly score is assigned to each node in the time series according to the difference between the reconstructed time series and the original time series, so as to comprehensively evaluate the time series anomaly status of each node;

[0019] According to the difference between the generated time series and the original time series, an anomaly score is assigned to each time node in the time series: the score of a normal node is 0, and the score of an abnormal node is 1.

[0020] Generally speaking, the present invention has the following beneficial effects:

[0021] 1. The method of the present invention designs an innovative framework that combines Informer with the denoising diffusion model for time series anomaly detection. This method can effectively handle the global dependencies, complex patterns and long-term relationships in time series data, and restore the time series data by gradually removing noise, thus significantly improving the model's ability to capture and identify anomalies; in addition, this combination method also accelerates the model training process and effectively alleviates the pressure on memory and other computing resources;

[0022] 2. The method of the present invention introduces an efficient ProbSparse self-attention mechanism to replace the traditional self-attention mechanism; this mechanism achieves a time complexity of O(LlogL) and memory usage efficiency, greatly improving the computing efficiency;

[0023] 3. The method of the present invention introduces a distillation self-attention mechanism, which reduces the total space complexity to O((2 - ∈)LlogL) by preferentially processing the dominant attention scores in the N-layer stacked structure, so as to more effectively process long sequence inputs. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flowchart of the method for time series anomaly detection based on multi-model fusion provided for the implementation of the present invention;

[0025] Figure 2 It is a framework diagram of the time series anomaly detection algorithm based on multi-model fusion provided for the implementation of the present invention;

[0026] Figure 3 It is the effect comparison of the method for time series anomaly detection based on multi-model fusion provided for the implementation of the present invention based on a public dataset; DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present invention.

[0028] The core of the present invention is to provide a time series anomaly detection method based on multi-model fusion, which can accurately detect anomalies, significantly reduce the running time, improve the utilization rate of computing resources, and thus can efficiently identify anomaly situations in a short time.

[0029] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0030] Figure 1 The flowchart of a time series anomaly detection method based on multi-model fusion provided for the embodiments of the present invention is as Figure 1 shown, including the following steps:

[0031] 101: Design a mask mechanism based on an autoencoder, reconstruct the time series through the autoencoder, and dynamically adjust the mask based on the error between the reconstructed time series and the original time series to generate a mask matrix;

[0032] Use the autoencoder to perform a preliminary reconstruction of the time series;

[0033] Calculate the reconstruction error: Each node is associated with a reconstruction error. According to the reconstruction error generated by the autoencoder and the actual anomaly ratio, determine which nodes need to be masked;

[0034] Calculate the multivariate similarity metric: Use the multivariate similarity metric between the masked nodes and their neighborhood nodes to determine the size of the mask matrix. If the multivariate similarity metric is greater than the threshold, the mask window expands in both directions, otherwise the expansion stops;

[0035] Generate the mask matrix: Generate the mask matrix M according to the reconstruction error and the required mask size. This matrix determines which nodes should be masked in the subsequent self-attention calculation process.

[0036] 102: Design a method that combines a denoising diffusion model and an Informer model. In this method, the denoising diffusion model sequentially adds noise in a Markov chain manner during the forward process and learns how to map back from the latent space to the signal space to remove this noise. Combining the Informer as the neural network structure inside the diffusion model enhances the processing ability of long sequence data and also accelerates the model convergence speed;

[0037] Denoising diffusion models represent a class of generative models that have demonstrated top - level performance in multiple fields such as image, audio, video data, etc.; denoising diffusion models regard the stochastic process as a partial differential equation and solve this equation numerically to obtain the probability distribution function of the stochastic process; as the noise gradually diffuses, the probability distribution function of the stochastic process also evolves, reflecting the underlying dynamic characteristics; denoising diffusion models consist of a forward diffusion process and a learnable inverse process. The forward diffusion process is denoted as q(x t |x t-1 ), which gradually transforms the data from the target distribution P(x0) into a Gaussian distribution by adding noise step by step; while the inverse process is denoted as p θ (x t-1 |x t ), and generates samples by converting the noise back to q(x0); in short, denoising diffusion models simulate the degradation process of data by adding noise to the data, and then train a model to learn how to recover the original data from the noisy data; this process can not only generate high - quality data samples, but also effectively capture the complex structure and dynamic changes of the data.

[0038] Replaced the U - Net inside the denoising diffusion model with Informer as the neural network architecture to better handle the deep local features and long - term dependencies in time - series data; the original denoising diffusion model used U - Net as the internal neural network to learn the inverse process; in this paper, we replaced the convolutional and pooling layers of U - Net in the denoising diffusion model with the ProbSpare self - attention mechanism and the distillation self - attention mechanism of Informer; this modification not only helps to capture the local and global relationships in time - series data, but also effectively reduces the memory consumption by reducing the time dimension of the feature map; at the same time, it optimizes the time complexity in the attention calculation process and speeds up the training speed.

[0039] ProbSpare self - attention mechanism: Although the traditional self - attention mechanism is powerful, it has problems of high computational complexity and large memory consumption when dealing with large - scale inputs; to alleviate these problems, the ProbSparse self - attention mechanism achieves sparsification by allowing each key to only focus on the most significant u queries, thus greatly reducing the computational amount and memory usage; the formula of the ProbSparse self - attention mechanism is as follows: where Q is a sparse matrix of the same size as q, only containing the Top - u queries under the sparsity measurement M(q i ,K); by controlling a constant sampling factor c, set u = c·ln L Q, which enables the ProbSparse self-attention mechanism to only compute O(ln L Q ) dot products during each query-key lookup, and the memory usage per layer remains at O(L K ln L Q );

[0040] In the multi-head perspective, this attention mechanism generates different sparse query-key pairs for each head, thus avoiding severe information loss; nonetheless, traversing all queries to compute the measurement M(q i ,K) still requires computing each dot product pair, i.e., with a complexity of O(L Q L K ); moreover, there may be numerical stability issues with the LSE operation; for this reason, we propose an empirical approximation method to efficiently obtain the query sparsity measurement;

[0041] Lemma 1: For each query q_i∈R^d and k_i∈R^d in the key set K, we have the following bounds:

[0042] Based on Lemma 1, we propose the maximum mean measurement: Under the long-tail distribution, we only need to randomly sample U = L K ln L Q dot product pairs to compute M(q i ,K), that is, filling other pairs with zeros; then, select the sparse Top-u as Q; the maximum operator is less sensitive to zero values and numerically stable in M(q i ,K); in fact, the input lengths of the query and the key are usually equal in the self-attention calculation, i.e., L Q = L K = L, so the total time complexity and space complexity of the ProbSparse self-attention mechanism are both O(LlnL);

[0043] Distilled self-attention mechanism: As a natural result of the ProbSparse self-attention mechanism, there are redundant value combinations V in the feature map of the encoder; to highlight the combinations with dominant features, we generate a focused self-attention feature map in the next layer; this process significantly reduces the time dimension of the input; our distillation process is passed from the j-th layer to the j + 1-th layer, with the formula as follows: where [·] ABrepresents an attention block, which contains multi-head ProbSparse self-attention and necessary operations; here, Conv1d(·) applies a 1D convolution filter (kernel width = 3) in the time dimension and uses the ELU activation function; we add a max pooling layer with a stride of 2 after stacking one layer, and convert X t Downsample to half, thus reducing the overall memory usage to O((2-∈)LlogL), where ∈ is a small number;

[0044] To make the distillation operation more robust, we build a copy of the main stack with half the input and gradually reduce the number of self-attention distillation layers, removing one layer at a time to form a pyramid-like structure so that the output dimensions are aligned; finally, we concatenate the outputs of all stacks to get the final hidden representation of the encoder;

[0045] By distilling the self-attention mechanism, we not only reduce the redundancy in the feature map, but also significantly reduce the memory usage and enhance the robustness of the model; this process achieves efficient feature extraction and compression through the combination of multi-head ProbSparse self-attention and 1D convolution, as well as the design of maximum pooling and pyramid structure;

[0046] In practical applications, before using the ProbSparse self-attention mechanism and distillation mechanism, the matrix mask obtained in step 102 is applied to the self-attention calculation to avoid leakage of abnormal information leading to reconstruction of abnormal nodes; then training is performed to reconstruct the original time series from the noisy time series, and then the noise added by the denoising diffusion model and the noise prediction obtained by the Informer model training are combined to obtain the reconstructed time series data.

[0047] 103: According to the difference between the generated time series and the original time series, an anomaly score is assigned to each node in the time series, thereby comprehensively evaluating the anomaly status of the time series of each node;

[0048] In the anomaly detection part, we use Kl divergence to quantify the difference between the reconstructed time series and the original time series, and calculate the anomaly score of each time node: the normal node score is 0, and the abnormal node score is 1; then we comprehensively evaluate the abnormal status of the time series of each node, so that we can accurately detect anomalies.

[0049] In order to further illustrate the practical effect of the time series anomaly detection method based on multi-model fusion provided by the present invention, the following is a specific Figure 3 To elaborate:

[0050] To comprehensively evaluate the performance of our research method, we not only conducted a detailed comparative analysis with three outstanding baseline models but also specifically examined the performance and training time of a model that only uses a single self-attention module;

[0051] The selected baseline models include: single-head self-attention model (Attention), deep autoencoding Gaussian mixture model (DAGMM), time series anomaly detection method based on correlation difference (Anomaly-Transformer), and graph deviation network (GDN).

[0052] Figure 3 Shows the comparison of our proposed research method with four baseline models in terms of accuracy, recall, F1 score, and training time on five different datasets. The best results are highlighted in bold, while the second-best results are underlined.

[0053] P, R, and F1 represent accuracy, recall, and F1 score (expressed as %), respectively. The F1 score is the harmonic mean of accuracy and recall. The higher the values of these three metrics, the better the performance of the model. Speed represents the training time (in seconds), and the smaller this metric is, the better.

[0054] The Attention model has an extremely short training time but performs poorly in terms of performance. This indicates that for application scenarios with extremely high real-time requirements and relatively low requirements for model accuracy, such as rapid prototyping or preliminary data analysis, this model may be a suitable choice. However, due to its performance limitations, it may not be suitable for tasks that require highly accurate predictions.

[0055] The DAGMM model combines the dimensionality reduction ability of the deep autoencoder and the density estimation ability of the Gaussian mixture model skillfully, achieving better performance than Attention while maintaining a relatively short training time. This feature makes DAGMM an ideal choice for projects that pursue higher performance while hoping to reduce computational resource consumption. For example, in resource-constrained edge computing environments, DAGMM can be an effective solution.

[0056] Anomaly-Transformer has slightly higher performance than our proposed method on some datasets because, under the same experimental environment, we increased the number of training epochs to improve the model performance as much as possible. Therefore, although the final performance gap is not large, Anomaly-Transformer has a longer training time. This shows that when there are extremely high requirements for model performance and longer training time can be accepted, Anomaly-Transformer is a good choice.

[0057] GDN also belongs to large models with relatively high computational costs. Its training time far exceeds that of lightweight models, but it has obvious performance advantages. This indicates that in application scenarios that pursue extreme performance, such as financial fraud detection and medical diagnosis, GDN can provide more reliable services.

[0058] In contrast, our proposed research method significantly shortens the training time while ensuring performance similar to or even better than Anomaly-Transformer and GDN, and effectively solves the problem of high memory consumption. This means that our method can not only demonstrate excellent performance in large-scale data processing tasks, but also greatly reduce the demand for computing resources, improving the efficiency of model training and deployment. This feature is particularly important for applications that require rapid response and efficient processing of large amounts of data, proving the feasibility and effectiveness of our method. Whether it is real-time anomaly detection in online services or complex pattern recognition in a big data environment, our method can provide an efficient and reliable solution.

[0059] The related technical solutions are the same as those in the above embodiments and will not be elaborated here.

[0060] The above has introduced in detail the time series anomaly detection method based on multi-model fusion provided by the present invention. The various embodiments in the specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other.

Claims

1. A time series anomaly detection method based on multi-model fusion, characterized in that: The following steps are involved: 101: Designed a masking mechanism based on an autoencoder to reconstruct the time series through the autoencoder, and dynamically adjusted the mask based on the error between the reconstructed time series and the original time series to generate a mask matrix; 102: Design a method that combines a denoising diffusion model and an informer model, where the forward diffusion process of the denoising diffusion model adds noise to the time series data, and the informer model is used to train a denoising function to reconstruct the original time series; 103: Reconstruct the time series using the above combined model, calculate the difference between the reconstructed time series and the original time series, assign an anomaly score to each time series node, and then comprehensively evaluate the anomaly status of the time series.

2. The time series anomaly detection method based on multi-model fusion according to claim 1 is characterized in that: In step 101: In order to solve the "self-information maximization problem" encountered by the traditional self-attention mechanism in time series anomaly detection, we proposed a masking mechanism based on an autoencoder; this method uses an autoencoder to preliminarily reconstruct the time series data to obtain the reconstruction error of each node, and determines the nodes that need to be masked based on these reconstruction errors and the actual anomaly ratio; Then, a multivariate similarity metric (including Pearson correlation coefficient, cosine similarity, etc.) is introduced to determine the size of the mask matrix. If the multivariate similarity metric is greater than the threshold, the mask window expands in two directions, otherwise it stops expanding. Thus, a mask matrix is ​​obtained to determine which nodes should be masked in the subsequent self-attention calculation process.

3. The time series anomaly detection method based on multi-model fusion according to claim 1 is characterized in that: In step 102: The denoising diffusion model represents a class of generative models that have demonstrated top performance in multiple fields such as image, audio, and video data. The denoising diffusion model regards the random process as a partial differential equation and solves the equation numerically to obtain the probability distribution function of the random process. As the noise gradually diffuses, the probability distribution function of the random process also evolves, reflecting the underlying dynamic characteristics. The denoising diffusion model consists of a forward diffusion process and a learnable inverse process. The forward diffusion process is denoted as q(x t |x t-1 ), which gradually transforms the data from the target distribution P(x0) into a Gaussian distribution by gradually adding noise; the inverse process is denoted by p θ (x t-1 |x t ), generate samples by converting the noise back to q(x0); in short, the denoising diffusion model simulates the data degradation process by adding noise to the data, and then trains a model to learn how to restore the original data from the noisy data; this process can not only generate high-quality data samples, but also effectively capture the complex structure and dynamic changes of the data; We replaced the U-Net network with Informer as the neural network architecture of the denoising diffusion model to better handle deep local features and long-term dependencies in time series data; the original denoising diffusion model used U-Net as the internal neural network to learn the inverse process; in this paper, we replaced the convolutional layers and pooling layers of U-Net in the denoising diffusion model with ProbSpare self-attention mechanism and distilled self-attention mechanism; this modification not only helps to capture local and global relationships in time series data, but also effectively reduces memory consumption by reducing the time dimension of feature maps; at the same time, it optimizes the time complexity of the attention calculation process and speeds up training; ProbSpare self-attention mechanism: Although the traditional self-attention mechanism is powerful, it has the problems of high computational complexity and high memory consumption when processing large-scale inputs. To alleviate these problems, the ProbSparse self-attention mechanism achieves sparsification by allowing each key to focus only on the most significant u queries, thereby greatly reducing the amount of computation and memory usage. The formula of the ProbSparse self-attention mechanism is as follows: where Q is a sparse matrix of the same size as q and contains only the sparsity measure M(q i ,K) under Top-u query; by controlling a constant sampling factor c, set u = c·ln L Q , which allows the ProbSparse self-attention mechanism to only compute O(lnL Q ) dot products, and the memory usage of each layer remains at O(L K l Q ); In the multi-head perspective, this attention mechanism generates different sparse query-key pairs for each head, thus avoiding severe information loss; nevertheless, it traverses all queries to calculate the measure M(q i ,K) still needs to calculate each dot product pair, that is, O(L Q L K ) complexity; In addition, LSE operations may have numerical stability issues; to this end, we propose an empirical approximation method to efficiently obtain query sparsity measurements; Lemma 1: For every query q i ∈R d and k in the key set K i ∈R d , we have the following bounds: When i ∈K, the formula also holds true; According to Lemma 1, we propose the maximum mean measure: Under the long-tail distribution, we only need to randomly sample U = L K l Q dot product pairs to calculate M(q i ,K), that is, fill the other pairs with zeros; then, select the sparse Top-u as Q; the maximum operator in M(q i ,K) is not very sensitive to zero values ​​and is numerically stable; in practice, the input lengths of the query and key are usually equal in the self-attention computation, i.e., L Q =L K =L, so the total time complexity and space complexity of the ProbSparse self-attention mechanism are both O(LlnL); Distillation self-attention mechanism: As a natural consequence of the ProbSparse self-attention mechanism, there are redundant value combinations V in the feature map of the encoder; in order to highlight the combination with dominant features, we generate a focused self-attention feature map in the next layer; this process significantly reduces the time dimension of the input; our distillation process is forwarded from the jth layer to the j+1th layer, and the formula is as follows: in[·] AB represents an attention block, which contains multi-head ProbSparse self-attention and necessary operations; here, Conv1d(·) applies a 1D convolution filter (kernel width = 3) in the time dimension and uses the ELU activation function; we add a max pooling layer with a stride of 2 after stacking one layer, and convert X t Downsample to half of it, thus reducing the overall memory usage to O((2-∈)L log L), where ∈ is a small number; To enhance the robustness of the distillation operation, we build a copy of the main stack, halve the input, and gradually reduce the number of self-attention distillation layers, removing one layer at a time to form a pyramid-like structure so that the output dimensions are aligned; Finally, we concatenate the outputs of all stacks to obtain the final hidden representation of the encoder; Through self-attention distillation, we not only reduce the redundancy in feature maps, but also significantly reduce memory usage and enhance the robustness of the model; this process achieves efficient feature extraction and compression through the combination of multi-head ProbSparse self-attention and 1D convolution, as well as the design of maximum pooling and pyramid structure; In practical applications, before using the ProbSparse self-attention mechanism and distillation mechanism, the matrix mask obtained in step 102 is applied to the self-attention calculation to avoid leakage of abnormal information leading to reconstruction of abnormal nodes; then training is performed to reconstruct the original time series from the noisy time series, and then the noise added by the denoising diffusion model and the noise prediction obtained by the Informer model training are combined to obtain the reconstructed time series data.

4. The time series anomaly detection method based on multi-model fusion according to claim 1 is characterized in that: In step 103: In the anomaly detection part, we use Kl divergence to quantify the difference between the reconstructed time series and the original time series, and calculate the anomaly score of each time node: the normal node score is 0, and the abnormal node score is 1; then we comprehensively evaluate the abnormal status of the time series of each node, so that we can accurately detect anomalies.

Images