Time sequence anomaly detection method and system based on frequency enhanced convolution Transform

Abstract

The invention belongs to the field of time sequence anomaly detection, and discloses a time sequence anomaly detection method and system based on frequency enhanced convolution Transform, and the method comprises the steps: carrying out the sequence-level preprocessing of an input sequence, including normalization and block generation; encoding the input sequence by adopting a multi-head self-attention mechanism; extracting interrelationships in the sequence by a full convolution module having a plurality of stacked 1D convolution layers of different kernel sizes; s2 and S3 are repeated, and multi-layer coding is carried out on the sequence; converting the encoded embedding into a frequency signal by using a Fourier transform method; the KL divergence and the absolute error are respectively used for calculating the consistency loss of the time domain and the frequency domain, calculating the point-by-point anomaly score, and identifying whether the point is abnormal or not. Wide experiments on five common data sets show that the method is superior to an existing method in the aspect of abnormal recognition.

Description

Technical Field

[0001] The present invention belongs to but is not limited to the technical field of time series anomaly detection, and in particular relates to a time series anomaly detection method and system based on frequency enhanced convolutional Transformer. Background Art

[0002] Time series anomaly detection has been widely used in fields such as water treatment equipment, aerospace, and servers. The purpose of time series anomaly detection is to identify instances with rare or abnormal patterns that have a negative impact on system operation. Effective detection of abnormal patterns is critical to preventing damage to the economy and infrastructure. For example, in the financial industry, monitoring and identifying abnormal fluctuations in financial data can help reduce unnecessary financial losses.

[0003] Traditional time series anomaly detection frameworks use statistical and classical machine learning methods, which can capture shallow representations or dependencies between sequences, but cannot model complex trends or discover abnormal latent discriminant spaces. Subsequently, due to its powerful ability to extract latent features, deep learning has gradually become one of the most effective learning tools, and many time series anomaly detection methods based on deep learning have been introduced and achieved impressive performance.

[0004] However, identifying anomalies in time series remains challenging. First, the definition of anomalies is often difficult to determine and standardize. For some periodic time series, irregular values ​​or patterns are considered anomalies, while for stable series, sudden changes or outliers are considered anomalies. Defining anomalies with high confidence requires extensive expert knowledge and experience, and is therefore difficult to achieve. Second, abnormal samples are rare in real-world applications, and most supervised or semi-supervised methods rely on labor-intensive label annotations. Third, the complexity of time series data makes the identification task more difficult. For example, the high dimensionality of time series indicates that there are complex spatial interactions between variables, while autocorrelation also reflects the uncertainty of changes over time.

[0005] Unsupervised learning methods can adaptively learn complex representations without labor-intensive label annotations, remain competitive in addressing these challenges, and are effective in the problem of time series anomaly detection. Researchers have utilized a variety of learning strategies to address these challenges. However, there is still some room for improvement. First, we note that a large portion of studies utilize reconstruction-based unsupervised learning methods to reconstruct the patterns of normal instances, while instances that fail to be reconstructed are considered anomalies. When combined with other machine learning methods, this strategy has a strong ability to model complex time series. However, the presence of anomalies in the sequence will limit the model's ability to capture the key features of the normal pattern. Anomalies in time series usually appear in the form of time dependencies rather than single points, which greatly increases the reconstruction loss and makes it difficult to learn the pure pattern of normal sequences. Second, Transofrmer-based methods have a strong ability to model long-range feature information. In the process of dependency modeling, anomalies contain a large number of local fine-grained features, which are inevitably diluted or even masked by long-term dependency feature information, resulting in very limited model performance. Third, most studies only use time domain information to model time series, ignoring frequency information. Several studies have adopted frequency domain time series modeling, and these pioneering works have shown that frequency analysis in time series modeling can bypass the complex autocorrelation in the time domain of the series, which helps to improve the modeling ability and identify abnormal observations.

[0006] In view of the above analysis, the technical problems that need to be solved urgently in the prior art are:

[0007] Traditional methods usually rely on reconstruction-based techniques, which are easily affected by abnormal sample errors and cannot learn pure normal sequence patterns; the long-term dependency of sequence features will mask local semantic information. The use of frequency domain analysis can be immune to the influence of complex autocorrelations in the sequence, but frequency domain analysis is often overlooked in anomaly detection.

[0008] Multi-source time series anomaly detection method and system based on parallel time convolution. This method designs a parallel convolution module to encode time series data, captures the relationship features in the time series data, and calculates the anomaly score of the time series through the reconstruction loss of local features and the prediction loss of global features, thereby judging its abnormal state. However, this method ignores the important information in the time-frequency domain characteristics. The frequency domain characteristics, as another important branch independent of the time domain characteristics, can reflect the important frequency composition relationship of the time series. In addition, the training of this method depends on the reconstruction loss, which is quite sensitive to abnormal data points, resulting in a decrease in training efficiency and accuracy.

[0009] Time series anomaly detection based on contrastive learning. This method uses a time encoder, a frequency encoder, and a residual encoder to capture different characteristics of the time series. This method can effectively model the normal pattern of the time series by using the difference between positive and negative sample pairs. However, due to the need to generate a large number of negative samples, the method has a high computational complexity and has certain shortcomings in terms of timeliness.

[0010] A Transformer-based multivariate time series anomaly detection method. This method uses the self-attention mechanism to extract the latent features of the time series, fuses the latent features with the position encoding, and reconstructs the time series using the encoder-decoder structure. However, Transformer has a strong ability to learn long-term features, but its ability to capture local features is insufficient, resulting in inaccurate local features, which affects the learning of sequence features. Summary of the invention

[0011] In view of the problems existing in the prior art, the present invention provides a time series anomaly detection method and system based on frequency enhanced convolutional Transformer.

[0012] The present invention is implemented as follows: a time series anomaly detection method based on frequency enhanced convolutional Transformer, characterized in that the time series anomaly detection method based on frequency enhanced convolutional Transformer specifically includes:

[0013] S1: Perform sequence-level preprocessing on the input sequence, including normalization and block generation;

[0014] S2: Use a multi-head self-attention mechanism to encode the input sequence;

[0015] S3: Extracting the mutual relations in the sequence through a fully convolutional module with multiple stacked 1D convolutional layers with different kernel sizes;

[0016] S4: Repeat S2 and S3 to perform multi-layer encoding on the sequence;

[0017] S5: Convert the encoded embedding into a frequency signal using a Fourier transform method;

[0018] S6: Use KL divergence and absolute error to calculate the consistency loss in the time domain and frequency domain respectively, calculate the point-by-point anomaly score, and identify whether the point is an anomaly.

[0019] Further, the S1 includes:

[0020] (1) Sequence normalization, the formula is as follows:

[0021]

[0022] Where Mean(·) and Var represent the mean function and variance function respectively, v i represents the i-th variable;

[0023] (2) Block generation, each time series Can be divided into a series of blocks, each block can be represented as Where P and N represent the block size and the number of blocks respectively.

[0024] Furthermore, S2 implements contrastive representation learning based on the transformer architecture, treating the inter-block level and the intra-block level as two different contrastive views:

[0025] (1) Inter-block view: considering the embedding of inter-block time series The model uses a self-attention mechanism to encode the input sequence. First, for each head, the self-attention mechanism weight matrix is ​​initialized:

[0026]

[0027] in Represents query and keyword respectively. Representative value, express The learnable weight matrix, h = 1, ..., H and H are the number of heads of most attention mechanisms. Then, the attention weights of different blocks are calculated.

[0028]

[0029] Where σ(·) is the softmax regularization function, which upsamples the attention weight matrix:

[0030]

[0031] Where Upsampling(·) represents the upsampling function, represents the inter-block-level extended attention weights, and then, the sequence encoded by the self-attention mechanism is calculated:

[0032]

[0033] Where ρ is the sigmoid activation function, Represents the embedded output of the block-based transformer. Finally, the model concatenates the multi-head representation:

[0034]

[0035] where ||(·) represents the concatenation function, represents the final intra-block representation;

[0036] (2) Intra-block view: Let the embedding representation of intra-block timing be For each header, initialize the query key-value matrix:

[0037]

[0038] in Represents query and keyword respectively. Representative value, express The learnable weight matrix is ​​then used to calculate the attention weights of different blocks:

[0039]

[0040] in The attention of different elements in a block is extended by upsampling:

[0041]

[0042] in represents the attention weights of intra-block level expansion, and then, the sequence encoded by the self-attention mechanism is calculated:

[0043]

[0044] in Represents the embedded output of the block-based transformer. Finally, the model concatenates the multi-head representation:

[0045]

[0046] in Represents the final inter-block representation.

[0047] Furthermore, S3 uses multi-scale parallel convolution operations to extract more high-order information, and its formula is as follows:

[0048]

[0049] Among them, Conv k Represents the convolution operation with a convolution kernel size of k, and then concatenates it into a high-order representation:

[0050]

[0051] Finally, multi-scale latent representations are fused through convolution transformation

[0052]

[0053] Further, in S5, a fast Fourier transform (FFT) algorithm is used to realize the transformation from the time domain to the frequency domain. The Fourier transform formula is:

[0054] f k (t) = exp(-j(2π / L)kt)

[0055]

[0056] According to the output of the convolution module and Get the frequency domain representation:

[0057]

[0058] in represents fast Fourier transform, FN represents Frequency domain information, F P express frequency domain information.

[0059] Further, in S6, in the abnormal reasoning stage, the test set is input into the trained model, and the abnormal score is calculated according to the consistency information from the two contrasting views, including the time domain consistency and the frequency domain consistency. The abnormal score in the time domain can be formulated as follows:

[0060]

[0061] The anomaly score in the frequency domain can be formulated as follows:

[0062]

[0063] The final scores are as follows:

[0064]

[0065] Where α is a hyperparameter, is the point-by-point anomaly score. The higher the value, the greater the inconsistency. The hyperparameter threshold ρ is used to identify whether the point is abnormal. If the calculated score exceeds the defined threshold ρ, the output For exception:

[0066]

[0067] Another object of the present invention is to provide a time series anomaly detection system based on frequency enhanced convolutional Transformer, the system specifically comprising:

[0068] Sequence-level preprocessing module for normalization and block generation;

[0069] The transformer module encodes the long-term dependencies of time series at both the inter-block level and the intra-block level;

[0070] Convolutional modules are used to enhance the fine-grained local details of dependencies and better extract the mutual relations in the sequence;

[0071] A Fourier transform module for converting the encoded embedding into a frequency signal;

[0072] Output module, used to output abnormal results.

[0073] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0074] First, we propose a novel frequency-enhanced convolutional transformer architecture for multivariate time series anomaly detection. The model exploits two different patch-based unsupervised contrastive learning views to model sequence invariants and amplify the differences between normal and anomalous instances. In addition, a fully convolutional module is introduced to enhance fine-grained local semantics.

[0075] This paper introduces a robust loss function based on KL divergence, which has only positive examples, for modeling dependencies in the time domain. This approach significantly reduces computational and space costs and overcomes the limitations of reconstruction loss.

[0076] In addition to the time domain analysis, the present invention also combines frequency analysis, which calculates the absolute error to measure the consistency in the frequency domain without being affected by autocorrelation, thereby improving the detection performance.

[0077] Second, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0078] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are as follows: In terms of expected benefits, the present invention can bring huge economic benefits to multiple industries, especially in the fields of water treatment equipment, aerospace, server monitoring, and financial risk control, and can effectively prevent equipment failures, safety accidents or financial losses caused by abnormal events. In terms of commercial value, the technical solution of the present invention can be embedded in the existing monitoring system to provide real-time and accurate anomaly detection services. This will provide value-added services to related companies and increase their market competitiveness. At the same time, the technology can also be used as an independent solution to provide customized services for companies that need anomaly detection services and open up new business areas. In addition, with the popularization of the Internet of Things and big data technologies, the acquisition and processing of time series data will become more and more convenient, and the technical solution of the present invention will have broader application prospects and greater commercial value.

[0079] (2) The technical solution of the present invention fills the technical gap in the industry at home and abroad: At present, although there are a variety of methods and technologies in the field of time series anomaly detection, traditional methods usually rely on reconstruction-based technologies, which are easily affected by abnormal sample errors and cannot learn pure normal sequence patterns. Although the Transformer-based method has a powerful ability to model long-range feature information, the long-term dependency of sequence features will obscure local semantic information. In addition, most studies only use time domain information to model time series and ignore frequency information. The technical solution of the present invention effectively solves these problems by combining a multi-head self-attention mechanism, a full convolution module, and frequency domain analysis, filling the technical gap in the field of time series anomaly detection at home and abroad.

[0080] (3) The technical solution of the present invention solves the technical problems that people have been eager to solve but have never been able to solve successfully:

[0081] Time series anomaly detection has always been a hot and difficult issue in the field of machine learning. People have always been eager to find a method that can accurately identify abnormal patterns and avoid the influence of abnormal sample errors. However, due to the complexity of time series data and the ambiguity of the definition of anomalies, this goal has never been achieved. The technical solution of the present invention successfully solves this technical problem by introducing technical means such as multi-head self-attention mechanism, full convolution module and frequency domain analysis. This method can accurately capture normal patterns in time series and effectively identify abnormal patterns, providing a new solution for the field of time series anomaly detection.

[0082] Third, the technical solution of the present invention addresses several problems existing in the existing time series anomaly detection technology and proposes a detection method based on frequency enhanced convolutional Transformer. When processing complex time series data, the existing technology often has the following technical problems:

[0083] 1. Insufficient feature extraction capabilities: Traditional time series anomaly detection methods usually rely on simple statistical features or linear models, which are difficult to capture complex patterns and nonlinear relationships in the sequence.

[0084] 2. Insufficient utilization of frequency domain information: Many existing technologies fail to fully utilize frequency domain features when analyzing time series, resulting in insufficient sensitivity in identifying potential abnormal patterns.

[0085] 3. Low computational efficiency: Some existing methods have low computational efficiency when processing large-scale time series data and cannot meet the needs of real-time detection.

[0086] The technical solution of the present invention effectively improves the ability of feature extraction by introducing a multi-head self-attention mechanism and stacked 1D convolutional layers, enabling the model to more deeply understand the complex relationships in time series data. Through Fourier transform, the method converts time domain signals into frequency domain signals, which helps to improve the sensitivity to abnormal patterns and thus achieve more accurate anomaly detection.

[0087] In practical applications, the detection method provided by the present invention has significant technical progress, which is specifically manifested in the following aspects:

[0088] 1. Improve the accuracy of anomaly detection: The frequency-enhanced convolutional Transformer can more comprehensively capture the abnormal features in the time series, significantly improving the accuracy of anomaly detection.

[0089] 2. Enhanced computational efficiency: The use of a fully convolutional structure and self-attention mechanism enables this method to have higher computational efficiency when processing large-scale data and meet the needs of real-time monitoring.

[0090] 3. Strong adaptability: This method can flexibly cope with various complex time series data, has good versatility, and can be widely used in financial monitoring, equipment failure prediction and other fields.

[0091] Through the above technical scheme, the present invention brings innovative solutions to the field of time series anomaly detection and promotes the progress and application of related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 It is a flow chart of a time series anomaly detection method based on frequency enhanced convolutional Transformer provided by an embodiment of the present invention;

[0093] Figure 2 is a sequence level preprocessing process diagram provided by an embodiment of the present invention;

[0094] Figure 3 It is an architecture diagram of a time series anomaly detection system based on frequency enhanced convolutional Transformer provided by an embodiment of the present invention;

[0095] Figure 4 is the experimental result of verifying the rationality of the loss function provided by the embodiment of the present invention;

[0096] Figure 5 is a sensitivity experiment result of block size provided by an embodiment of the present invention;

[0097] Figure 6 is the result of a sensitivity experiment on the window size provided by an embodiment of the present invention;

[0098] Figure 7is the F1 score of the parameter sensitivity test of embedded feature size, number of multi-head attention and number of model layers provided by the embodiment of the present invention;

[0099] Figure 8 is the efficiency comparison result provided by the embodiment of the present invention;

[0100] Fig. 9 It is a visualization of the inconsistency score difference provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0101] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0102] like Figure 1 As shown, an embodiment of the present invention provides a time series anomaly detection method based on frequency enhanced convolutional Transformer, and the method specifically includes:

[0103] S1: Perform sequence-level preprocessing on the input sequence, including normalization and block generation;

[0104] S2: Use a multi-head self-attention mechanism to encode the input sequence;

[0105] S3: Extracting the mutual relations in the sequence through a fully convolutional module with multiple stacked 1D convolutional layers with different kernel sizes;

[0106] S4: Repeat S2 and S3 to perform multi-layer encoding on the sequence;

[0107] S5: Convert the encoded embedding into a frequency signal using a Fourier transform method;

[0108] S6: Use KL divergence and absolute error to calculate the consistency loss in the time domain and frequency domain respectively, calculate the point-by-point anomaly score, and identify whether the point is an anomaly.

[0109] The embodiment of the present invention provides a time series anomaly detection method based on frequency enhanced convolutional transformer, which aims to improve the anomaly detection capability in time series data. The first step of the method is to perform sequence-level preprocessing on the input sequence. The preprocessing includes normalizing the time series data to eliminate the influence of different dimensions or amplitude differences on subsequent analysis. In addition, the time series is divided into several continuous small blocks through block generation technology to facilitate subsequent encoding and analysis.

[0110] Next, the method uses a multi-head self-attention mechanism to encode the preprocessed input sequence. The self-attention mechanism allows the model to pay attention to the correlation between different parts of the input sequence during the encoding process. Through the multi-head mechanism, the model can capture multiple relationships in the sequence in parallel, thereby improving the representation ability of complex sequences.

[0111] After that, the method extracts the mutual relations in the sequence through a fully convolutional module with multiple stacked 1D convolutional layers of different kernel sizes. The convolution operation can effectively capture local features, and through convolutional layers of different kernel sizes, the model can simultaneously obtain short-term and long-term dependencies in the sequence, thereby more comprehensively understanding the characteristics of the data.

[0112] After completing the convolutional feature extraction, the method repeats steps 2 and 3 multiple times to perform multi-layer encoding. Through multi-layer encoding, the model gradually builds a rich representation of the input sequence, which contains deep information of the time series and facilitates subsequent anomaly detection.

[0113] Next, the encoded embedding is converted into a frequency signal using the Fourier transform method. The representation in the frequency domain can help the model identify potential abnormal patterns, especially when certain frequency components in the time series data change suddenly, which are often closely related to abnormal events.

[0114] Finally, the method uses KL divergence and absolute error to calculate the consistency loss in the time domain and frequency domain respectively. Through these loss functions, the model can evaluate the consistency of the time series in the two domains and calculate the point-by-point anomaly score. Based on the calculation results, the model identifies which data points are abnormal, thereby completing the anomaly detection task of the time series.

[0115] S1, input data normalization helps to accelerate convergence and improve the stability of the deep learning process. Most Transformer-based frameworks normalize each timestamp and standardize the correlation between variables at a certain timestamp. When the event changes at that time point, the normalization of the timestamp will not only produce interactive uncertainty and interference in the delayed and non-causal process, but also lead to the problem of over-smoothing. Sequence-level normalization can handle the problem of non-stationary time series data, and its effectiveness has been successfully verified. In addition, sequence-level normalization can reduce the inconsistent differences in the range between variables while maintaining the correlation within the sequence. Unlike most previous work, the present invention realizes the normalization of the sequence, such as Figure 2 (a), the formula is as follows:

[0116]

[0117] Where Mean(·) and Var represent the mean function and variance function respectively, v i represents the i-th variable.

[0118] In order to generate different contrasting views and capture potential dependency information from different views, the input sequence is split into different small blocks, which is an effective way to simplify the learning of dependency representations based on the Transformer architecture based on the self-attention mechanism. Figure 2 (b), each time series Can be divided into a series of blocks, each block can be represented as Where P and N represent the block size and number of blocks respectively. Generally speaking, the multivariate input sequence Can be divided into

[0119] The model then combines the batch size with the feature dimension, and the input sequence can be simplified to

[0120]

[0121] As mentioned in S2, in order to model the normal pattern of time series, we implement contrastive representation learning based on the transformer architecture. We regard the inter-block level and intra-block level as two different contrastive views.

[0122] For the inter-block view, each individual block is treated as a discrete unit, enabling a structured approach to analyze its different features, and the relationships and dependencies between these blocks can be effectively modeled using a multi-head self-attention mechanism. The model uses a self-attention mechanism to encode the input sequence. First, for each head, initialize the self-attention mechanism weight matrix:

[0123]

[0124] in Represents query and keyword respectively. Representative value, express The learnable weight matrix, h = 1, ..., HandH is the number of heads of most attention mechanisms. Then, the attention weights of different blocks are calculated

[0125]

[0126] where σ(·) is the softmax regularization function. Note that as the block operation benefits from local information, the inter-block level ignores the correlation of elements in a single block. In other words, Unable to Multiply. Therefore, to calculate The attention weight matrix needs to be upsampled:

[0127]

[0128] Where Upsampling(·) represents the upsampling function, represents the attention weights of inter-block level expansion. Then, the sequence encoded by the self-attention mechanism is calculated:

[0129]

[0130] Where ρ is the sigmoid activation function, Represents the embedded output of the block-based transformer. Finally, the model concatenates the multi-head representation:

[0131]

[0132] where ||(·) represents the concatenation function, Represents the final intra-block representation.

[0133] Similar to the inter-block view, the self-attention mechanism is applied to the number of blocks N, and the weight matrix is ​​shared in the inter-block view and the inter-block view. Specifically, let the embedding representation of the intra-block sequence be For each header, initialize the query key-value matrix:

[0134]

[0135] in Represents query and keyword respectively. Representative value, express Then, the attention weights of different blocks are calculated:

[0136]

[0137] in Attention of different elements within a block. Since the intra-block level representation ignores the association between different blocks, the model uses upsampling to expand the intra-block attention:

[0138]

[0139] in represents the attention weight of intra-block level expansion. Then, the sequence encoded by the self-attention mechanism is calculated:

[0140]

[0141] in Represents the embedded output of the block-based transformer. Finally, the model concatenates the multi-head representation:

[0142]

[0143] in Represents the final inter-block representation.

[0144] One of the most prominent features of abnormal occurrence in time series is localization, which places higher requirements on the algorithm to capture local representations. Convolutional networks can not only capture global features, but also have the ability to sensitively reconstruct local features. The combination of Transformer and convolutional networks (such as Conformer) has shown great information extraction capabilities in many fields (such as 3D object detection). In order to establish a long-term interactive fusion of detailed local semantic information and extract additional latent representations to detect abnormal patterns, S3, considers a block-based Transformer encoding representation under a single contrast view For simplicity, the subscripts are omitted in the symbols. The model uses multi-scale parallel convolution operations to extract more high-order information. The formula is as follows:

[0145]

[0146] Among them, Conv k Represents the convolution operation with a convolution kernel size of k. Then concatenate into a high-level representation:

[0147]

[0148] Finally, multi-scale latent representations are fused through convolution transformation

[0149]

[0150] Through the convolution module, the model can obtain multi-scale potential representations with rich fine-grained local information.

[0151] and They are used for inter-block comparison view and intra-block comparison view respectively.

[0152] Although the Transformer architecture can dynamically evaluate the autocorrelation in the time series, it cannot capture the dependency characteristics beyond the autocorrelation. Frequency analysis, as an advanced time series modeling paradigm, can bypass the dependency caused by time domain autocorrelation and has great potential in modeling the normal pattern of time series. In order to enhance the modeling ability of the normal pattern of time series, the S5 is based on Fourier transform to obtain the representation in the frequency domain. Specifically, the fast Fourier transform (FFT) algorithm is used to realize the transformation from time domain to frequency domain. The Fourier transform formula is:

[0153] f k (t) = exp(-j(2π / L)kt)

[0154]

[0155] According to the output of the convolution module and Get the frequency domain representation:

[0156]

[0157] in represents fast Fourier transform, FN represents Frequency domain information, F P express Therefore, the contrast loss of frequency domain information consistency can be calculated by the following formula:

[0158]

[0159] where |(·)| is the modulus computation function and J represents the number of terms in the frequency domain representation. Note that different frequency components vary greatly in amplitude, which ultimately leads to unstable performance. Therefore, we utilize absolute loss instead of squared loss.

[0160] Through two contrasting channels, inter-block and intra-block representations are derived. In order to model the normal mode of the time series, the Kullback-Leibler (KL) divergence is used to evaluate the consistency between the representations from the two contrasting views. The motivation is that abnormal samples are very rare, and the encoding embeddings of normal samples from different views should remain consistent and similar in the latent space.

[0161] Specifically, the loss function can be formulated as follows:

[0162]

[0163] in are the multi-scale representations encoded by the convolutional module from the inter-block view and the intra-block view, respectively, KL(·) is the KL divergence function, and Ω(·) indicates the stop gradient update operation. Therefore, the loss function in the time domain can be expressed as:

[0164]

[0165] Where len is the number of channels for different block sizes.

[0166] Stopping the gradient operation helps the model get rid of trivial solutions, which is successfully demonstrated in the performance of model training. In addition, unlike traditional contrastive methods, we only use positive samples to implement contrastive learning. This strategy helps to improve efficiency without losing performance and reduce computational complexity. The overall loss function of the model is as follows:

[0167]

[0168] Where α is a hyperparameter used to balance and

[0169] In S6, in the abnormal reasoning stage, the test set is input into the trained model, and the abnormal score is calculated based on the consistency information from the two contrasting views, including the consistency in the time domain and the consistency in the frequency domain. The abnormal score in the time domain can be formulated as follows:

[0170]

[0171] The anomaly score in the frequency domain can be formulated as follows:

[0172]

[0173] The final scores are as follows:

[0174]

[0175] where α is a hyperparameter.

[0176] is the point-by-point anomaly score, and higher values ​​indicate greater inconsistency. In order to distinguish between normal and abnormal samples, the hyperparameter threshold ρ is used to identify whether the point is abnormal. If the calculated score exceeds the defined threshold ρ, the output is abnormal.

[0177]

[0178] like Figure 3 As shown, a time series anomaly detection system based on frequency enhanced convolutional Transformer provided by an embodiment of the present invention specifically includes:

[0179] Sequence-level preprocessing module for normalization and block generation. Sequence-level normalization improves the stability of the sequence and helps to capture potential dependencies more effectively. Block generation divides the sequence into smaller blocks to enhance the representation of local semantic information.

[0180] The transformer module encodes the long-term dependencies of time series at both the inter-block level and the intra-block level;

[0181] Convolutional modules are used to enhance the fine-grained local details of dependencies and better extract the mutual relations in the sequence;

[0182] A Fourier transform module for converting the encoded embedding into a frequency signal;

[0183] Output module, used to output abnormal results.

[0184] Relevant evidence of the technical effects achieved by the embodiments of the present invention.

[0185] (I) Dataset

[0186] The data experiment is conducted using five publicly available datasets. The statistics of the datasets are shown in the table. The dimension represents the recorded data at each timestamp; the training and test represent the number of training sets and test sets; the anomaly rate reflects the proportion of abnormal samples.

[0187] Table 1 Dataset

[0188]

[0189]

[0190] MSL: NASA's Mars Science Laboratory (SML) dataset shows a variety of information from sensors and actuators in the Mars rovers.

[0191] SMAP: The Soil Moisture Active-Passive (SMAP) dataset represents satellite-collected information on Earth's soil moisture, collected every two days.

[0192] PSM: Pooled Server Metrics (PSM) dataset is obtained from an e-commerce platform named eBay.

[0193] SMD: The Server Machine Dataset (SMD) dataset is obtained based on 28 different types of machines.

[0194] SWaT: The Safe Water as Treatment (SWaT) dataset is extracted from infrastructure water treatment systems.

[0195] 2. Baseline

[0196] We selected 11 algorithms as baselines to verify the effectiveness of the present invention, of which LOF and DAGMM are statistical or traditional machine learning anomaly detection methods, and the rest can be identified as deep learning methods.

[0197] LOF: This is an anomaly detection method based on local density deviation.

[0198] VAE: This is a stochastic variational inference and learning algorithm that works even in tricky situations.

[0199] DAGMM: This is an anomaly detection framework using normal patterns of time series based on Gaussian mixture models.

[0200] OmniAnomaly: This is a framework based on randomized recurrent neural networks to identify anomalies based on reconstruction.

[0201] InterFusion: This is an anomaly detection framework based on hierarchical variational autoencoders.

[0202] TS-CP2: It utilizes the contrastive learning paradigm based on change point detection for time series anomaly detection.

[0203] TranAD: It is a transformer-based framework that exploits the self-attention mechanism to model temporal dependencies in time series.

[0204] AnomalyTrans: It builds correlation differences by leveraging the transformer framework for time series anomaly detection.

[0205] DCFF-MTAD: Leveraging graph attention networks, this innovative two-channel approach aims to detect anomalies in multivariate time series data.

[0206] MAUT: This is a U-shaped transformer framework based on memory units to learn normal patterns of time series.

[0207] ATF-UAD: This is a reconstruction-based anomaly detection method that decomposes the time series into time and frequency domains.

[0208] 3. Evaluation indicators

[0209] When evaluating the performance of experiments, we rely on three key metrics: precision (P), recall (R), and F1 score (F1). These metrics help evaluate the accuracy and completeness of the results. Precision represents the proportion of correctly identified instances among all identified instances, providing insight into the reliability of the results. Recall, on the other hand, measures the proportion of correctly identified instances among all instances that should have been identified, providing a glimpse into the comprehensiveness of the results. The F1 score combines precision and recall to provide a comprehensive assessment of experimental performance. By utilizing these metrics, researchers can ensure that their results are both accurate and comprehensive, allowing them to draw meaningful conclusions from their experiments. The formulas for P, R, and F1 are as follows:

[0210]

[0211] Among them, TP, FP and FN represent true positive samples, false positive samples and false negative samples respectively.

[0212] In practical applications, anomalies usually appear continuously, and point anomalies cannot provide very valuable reference information. Next, we adopt the point adjustment method to perform experiments. If any point is identified as an anomaly within a segment, the segment will be marked as an anomaly.

[0213] 4. Experimental setup and implementation

[0214] The experiments were conducted using pyTorch in Python 3.9.12, deploying an NVIDIA A40 GPU, 40GB RAM, and 2.60GHz Xeon(R) Gold 6240 CPU. For the baseline method, we replicated it according to the source code provided by the authors.

[0215] The present invention includes three encoder layers. The hidden size is set to 128 and the number of attention heads is 1. Adam Optimizer is deployed and the learning rate is set to 10^(-4). More detailed experimental settings are shown in the table.

[0216] Table 2 Experimental settings on different benchmark datasets

[0217] Dataset Patchsize Patchsize Windowsize epochs MSL [3,5] 64 90 3 SMAP [3,5,7] 256 105 3 PSM [1,3,5] 256 60 3 SMD [5,7] 256 105 2 SWaT [3,5,7] 128 105 3

[0218] (V) Experimental results

[0219] We first evaluated the performance of the invention and the results are shown in the following table. The best result is highlighted in bold and the second place result is underlined. (P&R&F1)

[0220] Table 3 Experimental results

[0221]

[0222]

[0223] For traditional methods, DAGMM outperforms LOF on all five datasets due to its powerful ability to model time series distributions by stacking linear modules.

[0224] The deep learning baselines perform relatively better than traditional methods. Specifically, VAE optimizes the encoding representation according to the variational lower bound, and TS-CP2 identifies anomalies in time series based on change point detection. Therefore, they achieve better performance than traditional methods. OmniAnomaly utilizes RNN to learn dependencies in time series, while ATF-UAD reconstructs time series from the perspective of frequency and time domains. They are both able to extract trends in time series, thus significantly improving the performance. InterFusion deploys a variational autoencoder to extract features in normal time series and achieves stable performance using F1 measure on all datasets over 0.8. For the transformer-based frameworks, namely NormFAAE, MAUT, TranAD, and AnomalyTrans, they utilize self-attention mechanisms to understand the potential dependencies in time series and manage to reconstruct the normal patterns of time series. The average F1 values ​​of NormFAAE, MAUT, TranAD, and AnomalyTrans on the five datasets are 0.8812, 0.9090, 0.8891, and 0.9412, respectively, which is an impressive observation. DCFF-MTAD relies on graph neural networks and frequency information to capture spatiotemporal information from time series, showing non-trivial and competitive observations.

[0225] The present invention performs best among most baselines, which shows the effectiveness of the present invention. The present invention improves the F1 scores of MSL, SMAP, SWaT and PSM datasets by 1.01%, 0.1%, 1.98% and 1.83% respectively. This can be attributed to the fact that the present invention is able to extract representative dependencies of sequences based on the Transformer architecture and grasp local detailed semantics using convolutional modules from two contrasting views, which provides insightful guidance for sequence learning. In addition, in addition to the autocorrelation of time series, frequency signals also provide valuable information and help learn different representations from the frequency domain. Therefore, the present invention can understand and simulate the normal patterns of time series. However, on the SMD dataset, the present invention did not show amazing performance, which can be attributed to the challenging learning task of dependent features in SMD time series. Nevertheless, compared with most baselines, the present invention still has considerable observations on the SMD dataset.

[0226] (VI) Ablation experiment

[0227] We first verify the effectiveness of the stopping gradient strategy on the loss function. Table 4 shows the observed results. The best results are in bold. Specifically, if the stopping gradient is not implemented, the present invention still does not become a trivial solution and has competitive performance compared with the baselines. When the stopping gradient strategy is implemented for a single contrast view, the performance of MSL and PSM drops significantly, but still maintains a leading position compared with some baselines. When the stopping gradient strategy is implemented for two contrast views, the performance is the best, which proves the practicality of the stopping gradient strategy.

[0228] Table 4 Results of stopping gradient ablation experiment (P&R&F1)

[0229]

[0230]

[0231] We then designed three variants, FreCT norm , FreCT conv and FreCT fre , respectively represent the present invention without normalization operation, the present invention without convolution module and the present invention without frequency domain enhancement, to verify the effectiveness of normalization operation, convolution module and frequency enhancement learning module. The results are shown in the figure. The best result is highlighted in bold. From the results, it can be seen that the proposed method is similar to FreCT. norm , FreCT conv andFreCT fre , compared with the proposed method, the best performance is obtained, which proves the role of these modules. norm , FreCT conv and FreCT fre The average performance degradation on the five datasets is 1.79%, 1.26%, and 0.97% respectively. norm , the normalization operation helps to highlight the obvious features in the time series and makes the distribution easier to learn. conv , the convolution module helps to extract more interactive information in the sequence. fre ,The frequency enhanced learning module that resists autocorrelation can capture valuable information from the frequency domain for modeling the normal patterns of the sequence.

[0232] Table 5 Module ablation experiment results (P&R&F1)

[0233]

[0234]

[0235] (VII) Reasonableness Verification

[0236] We first verify the rationality of the loss function. We design several variants, namely the present invention with JS divergence and the present invention with asymmetric KL divergence, to prove the effectiveness of the present invention with symmetric KL divergence. Figure 4 The performance of the loss function rationality verification experiment is shown. The F1 and ACC metrics of the present invention with three loss functions exceed 0.8 and 0.94 respectively, which shows that our present invention has non-trivial solutions according to different optimization strategies. Specifically, compared with the present invention with KL divergence, the present invention with JS divergence performs relatively poorly. In addition, the performance of the present invention with asymmetric KL is not as good as that of the present invention, which shows that the symmetric KL divergence is worse than the asymmetric KL divergence in measuring the consistency between two different distributions.

[0237] Then we verified the rationality of the frequency transformation method. The transformation is achieved by mapping the time series to a set of mutually orthogonal bases. The choice of frequency transformation method depends on the analysis of the specific problem. We introduced several variants of FFT, which are based on mutually orthogonal sine functions to analyze signals. The observed results are shown in the table, where rfft calculates the one-dimensional Fourier transform of the real-valued input sequence, rfft-2d calculates the two-dimensional discrete Fourier transform of the real-valued input sequence, and FFT represents the original FFT introduced in the article.

[0238] Table 6 Results of reasonable verification of the frequency conversion method

[0239]

[0240]

[0241] By comparing different Fourier transform methods, we can observe that FFT achieves the best performance, which justifies the chosen frequency transform method. Note that although RFFT and RFFT-2D do not achieve the best performance, they still reach non-trivial solutions, which shows that our method can extract representative features for the anomaly detection problem.

[0242] 8. Hyperparameter Sensitivity

[0243] First, we test the effect of block size on anomaly detection performance. By using block sizes of different scales, we can obtain multi-scale block information for comparative learning. Figure 5As shown, the horizontal coordinate represents a set of different block sizes. For example, [1,3] means that we split the time series according to block size 1 and block size 3 respectively, and then the model learns separate representations for each block size channel, and finally combines these representations together. Based on the observations, some conclusions can be drawn. First, the block size has a significant impact on the performance of the present invention. For the dataset PSM, the optimal block size is [1,5]. However, when the block size is [1,3], the performance drops significantly. Second, the best choice of block size depends on the dataset. For the SMD dataset, the best choice of block size is [1,3], while for the SWaT dataset, the best choice is [3,5]. Third, the impact of block size on different datasets varies. For example, for the MSL dataset, the best choice is [3,5], while for SMAP, the best choice is [1,5].

[0244] The time window size is an important hyperparameter in time series anomaly detection, which determines the length of an instance. We have conducted extensive research on the time window size [30, 150] and observed that Figure 6 As shown in Figure 2, it is observed that the performance is relatively stable and superior within the time window size range [90, 120]. Small window sizes cannot contain enough sequence information, and large window sizes will introduce interference and noise, hindering feature learning.

[0245] In addition, we conducted sensitivity experiments on the number of layers, embedding size, and number of heads; the results are shown in Figure 7 As shown. Figure 7 (a) It can be seen that when the number of layers is 3, the performance of the present invention is the best, which indicates that the number of layers also affects the ability of the present invention to learn potential representations. Figure 7 (b) shows the effect of embedding size on the performance of the present invention. Generally speaking, a smaller embedding size will lead to insufficient feature representation, while a larger embedding size will lead to overfitting problems. Figure 7 (c) It can be seen that the performance is best when the number of heads is 1. Too many multi-head attention heads will make the representation insufficient for distinction.

[0246] 9. Efficiency

[0247] Our goal is to detect anomalies in time series. Efficient and timely detection helps reduce the losses caused by anomalies. To evaluate the efficiency of the present invention, we compared the training time of the present invention with several selected state-of-the-art algorithms and observed the following results: Figure 8 As shown. We select traditional machine learning and statistical methods and some relatively superior methods to replace all baselines, which can be regarded as the upper limit of all baselines. Therefore, the selected algorithms are representative in the time efficiency experiment, and the efficiency performance of the present invention is illustrated by comparing with the present invention.

[0248] From the observation results, the time required for training of the present invention is shorter than that of TranAD and AnomalyTrans on most data sets, indicating that dual-channel contrast consistency information learning is more effective than associated difference fitting learning and dual autoencoder architecture. In addition, the use of CNN to extract latent embedding makes the features more representative and discriminative, improving the recognition efficiency, while the training of the present invention takes more time than DAGMM and VAE. The first reason is that DAGMM is a traditional method without too many parameters to learn, while VAE is a lightweight base model with fewer parameters than the most advanced algorithms. The second reason is that they have limited ability to learn latent and discriminative representations, so they take less time to train.

[0249] In summary, the dosage of the present invention proposed by us is not only very competitive, but also has relatively superior efficiency.

[0250] 10. Visualization

[0251] To intuitively demonstrate the effectiveness of our invention in representation learning, we visualize the difference in the inconsistency score of each sample calculated by equation [Total Anomaly Score], as Fig. 9 By observing the distribution differences, we can see the differences in anomaly scores between samples, which can be used as a reference for anomaly detection.

[0252] Specifically, for all datasets, most scores are close to 0, which means that most contrastive representations of normal samples maintain a high degree of distribution consistency. In terms of outliers, for the MSL dataset, the outlier value is close to 0.8; for the SMAP, PSM and SWaT datasets, the outlier value is close to 1, and for the SMD dataset, the outlier value is close to 0.95. Therefore, the model can adaptively learn the inconsistent information of two contrasting views, and higher inconsistency scores can be regarded as anomalies.

[0253] There is also a plausible explanation for the observation that the performance on the SMD dataset is not as good as on other datasets. Fig. 9 (d), the distribution symmetry of most samples with scores close to 0 is not as good as that on other datasets. This phenomenon indicates that there is some noise, resulting in a small distribution tail. Other datasets, such as PSM and MSL, have relatively positive distribution symmetry, which is more discriminative for anomaly detection.

[0254] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0255] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A time series anomaly detection method based on frequency enhanced convolutional Transformer, characterized in that: The method specifically includes: S1: Perform sequence-level preprocessing on the input sequence, including normalization and block generation; S2: Use a multi-head self-attention mechanism to encode the input sequence; S3: Extracting the mutual relations in the sequence through a fully convolutional module with multiple stacked 1D convolutional layers with different kernel sizes; S4: Repeat S2 and S3 to perform multi-layer encoding on the sequence; S5: Convert the encoded embedding into a frequency signal using a Fourier transform method; S6: Use KL divergence and absolute error to calculate the consistency loss in the time domain and frequency domain respectively, calculate the point-by-point anomaly score, and identify whether the point is an anomaly.

2. The time series anomaly detection method based on frequency enhanced convolutional Transformer as claimed in claim 1, characterized in that: Said S1 comprises: (2) Sequence normalization, the formula is as follows: Where Mean(·) and Var represent the mean function and variance function respectively, v i represents the i-th variable; (2) Block generation, each time series Can be divided into a series of blocks, each block can be represented as Where P and N represent the block size and the number of blocks respectively.

3. The time series anomaly detection method based on frequency enhanced convolutional Transformer as claimed in claim 1, characterized in that: The S2 implements contrastive representation learning based on the transformer architecture, treating the inter-block level and intra-block level as two different contrastive views: (1) Inter-block view: considering the embedding of inter-block time series The model uses a self-attention mechanism to encode the input sequence. First, for each head, the self-attention mechanism weight matrix is ​​initialized: in Represents query and keyword respectively, Representative value, express The learnable weight matrix, h = 1, ..., H and H are the number of heads of most attention mechanisms. Then, the attention weights of different blocks are calculated. Where σ(·) is the softmax regularization function, which performs an upsampling operation on the attention weight matrix: Where Upsampling(·) represents the upsampling function, represents the attention weights of inter-block level expansion, and then, the sequence encoded by the self-attention mechanism is calculated: Where ρ is the sigmoid activation function, Represents the embedded output of the block-based transformer. Finally, the model concatenates the multi-head representation: where ||(·) represents the concatenation function, represents the final intra-block representation; (2) Intra-block view: Let the embedding representation of intra-block timing be For each header, initialize the query key-value matrix: in Represents query and keyword respectively, Representative value, express The learnable weight matrix is ​​then used to calculate the attention weights of different blocks: in The attention of different elements in a block is extended by upsampling: in represents the attention weights of intra-block level expansion, and then, the sequence encoded by the self-attention mechanism is calculated: in Represents the embedded output of the block-based transformer. Finally, the model concatenates the multi-head representation: in Represents the final inter-block representation.

4. The time series anomaly detection method based on frequency enhanced convolutional Transformer as claimed in claim 1, characterized in that: The S3 uses multi-scale parallel convolution operations to extract more high-order information, and its formula is as follows: z1=Conv1(z) z3=Conv3(z) z5=Conv5(z) Among them, Conv k Represents the convolution operation with a convolution kernel size of k, and then concatenates it into a high-order representation: Finally, the multi-scale latent representation z′ is fused through convolution transformation:

5. The time series anomaly detection method based on frequency enhanced convolutional Transformer as claimed in claim 1, characterized in that: The S5 uses a fast Fourier transform (FFT) algorithm to achieve the transformation from the time domain to the frequency domain. The Fourier transform formula is: f k (t)=exp(-j(2π / L)kt) According to the output z of the convolution module N ′ and z P ′Get frequency domain representation: in represents fast Fourier transform, F N Indicates z N ′’s frequency domain information, F P Indicates z P ′’s frequency domain information.

6. The time series anomaly detection method based on frequency enhanced convolutional Transformer as claimed in claim 1, characterized in that: In S6, in the abnormal reasoning stage, the test set is input into the trained model, and the abnormal score is calculated based on the consistency information from the two contrasting views, including the consistency in the time domain and the consistency in the frequency domain. The abnormal score in the time domain can be formulated as follows: Score tim (x)=∑KL(z P ',Ω(z N ')))+KL(Ω(z P '),With N ') The anomaly score in the frequency domain can be formulated as follows: The final scores are as follows: Score(x)=αScore tim (x)+(1-α)Score fre (x) Where α is a hyperparameter, Score(x) is the point-by-point anomaly score, and the higher the value, the greater the inconsistency. The hyperparameter threshold ρ is used to identify whether the point is an anomaly. If the calculated score exceeds the defined threshold ρ, the output y is an anomaly:

7. A time series anomaly detection system based on the frequency enhanced convolutional Transformer as described in claims 1-6, characterized in that: The system specifically includes: Sequence-level preprocessing module for normalization and block generation; The transformer module encodes the long-term dependencies of time series at both the inter-block level and the intra-block level; Convolutional modules are used to enhance the fine-grained local details of dependencies and better extract the mutual relations in the sequence; A Fourier transform module for converting the encoded embedding into a frequency signal; Output module, used to output abnormal results.

Images