Unsupervised time sequence anomaly detection method based on multi-scale reconstruction

Abstract

An unsupervised time series anomaly detection method based on multi-scale reconstruction processes a time series anomaly detection task from the aspect of different scales of a time series, and comprises the following steps: firstly, inputting time series data into a multi-scale decoupling module (MTD) to separate the time series data into components with different scales; and the reconstruction process is modeled as parallel full-scale time sequence reconstruction. This is because the normal time point and the abnormal time point have significant differences in different scales. In consideration of the correlation among different scales, the joint weights of different generators in the network are controlled by adaptively calculating the joint weights by reconstructing a residual error collaborative learning module (RRCL), so that interaction among different scales is realized. The model is designed, and multi-scale features of the time sequence are deeply extracted on the premise that an abnormal sample is not needed, so that the accuracy of time sequence anomaly detection is improved, and the robustness of the model is improved.

Description

Technical Field

[0001] This invention belongs to the fields of adversarial generative networks, residual neural networks, and time series anomaly detection, and in particular relates to a method for improving the performance of time series anomaly detection by utilizing multi-scale decoupling and reconstruction residual learning over time. Background Technology

[0002] Time series anomaly detection has wide applications in practical fields, such as industrial equipment condition monitoring, financial fraud detection, fault diagnosis, and routine monitoring and maintenance of automobiles. With the rapid development of various sensor technologies, a large amount of time series data is collected during the operation of different application systems. To ensure safety and prevent economic losses, effectively identifying abnormal patterns in the system is crucial. For example, in the security field, timely detection of anomalies during vehicle operation can prevent catastrophic failures and personal injury; in the financial field, detecting fraudulent activities is very important for reducing economic losses.

[0003] In anomaly detection, acquiring anomalous samples and detecting new anomalies are time-consuming and expensive, forcing us to develop unsupervised methods for more practical applications. Unsupervised time series anomaly detection is particularly challenging in practice. The model should learn information representations from complex temporal dynamics through unsupervised tasks. Nevertheless, it should also derive a distinguishable criterion capable of detecting rare anomalies from a large number of normal time points. Various classic anomaly detection methods provide many unsupervised paradigms, such as the density estimation method (LOF) proposed in the local outlier factor (Reference [1] Breunig M M, Kriegel HP, Ng RT, et al. LOF: identifying density-based local outliers [C] / / Proceedings of the 2000ACM SIGMOD international conference on Management of data. 2000: 93-104..), a class of support vector machines (OC-SVM) (Reference [2] Chen Z, Chen D, Zhang X, et al. Learning graph structures with transformer for multivariate time-series anomaly detection in IoT [J]. IEEE Internet of Things Journal, 2021, 9(12): 9179-9189.) and SVDD (Reference [3] Tax DMJ, Duin RP W. Support vector data description [J]. Machine learning, 2004, 54: 45-66). These classic methods do not take time information into account and are difficult to generalize to unseen real-world scenarios.Thanks to the representation learning ability of neural networks, recent deep models (Reference [4]: ​​Xu J, Wu H, Wang J, et al. Anomaly transformer: Time series anomaly detection with association discrepancy[J]. arxiv preprint arxiv:2110.02642,2021. Reference [5]: Yang Y, Zhang C, Zhou T, et al. Dcdetector: Dual attention contrastive representation learning for time series anomaly detection[C] / / Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and DataMining.2023:3033-3045. Reference [6]: Liu J, Chen S. Timesurl: Self-supervised contrastive learning for universal time series representation learning[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(12):13918-13926.) achieved better performance. One major class of methods is based on explicit association modeling for anomaly detection. Vector autoregression and state-space models belong to this class. Another class focuses on learning pointwise representations through carefully designed recurrent networks and self-supervised through reconstruction or autoregressive tasks. In general, these classical methods have difficulty learning information representations and modeling fine-grained associations. Recently, in deep learning models, decoupling time series into multiple independent components helps the model better understand and analyze its intrinsic characteristics.N-BEATS, N-HiTS, PatchTS and ETSformer (Reference [7]: ORESHKIN BorisN, CARPOV D, CHAPADOS N, et al. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. [J]. Learning, Learning, 2019. Reference [8]: CHALLU C, OLIVARES KinG, ORESHKIN BorisN, et al.N-HiTS: NeuralHierarchical Interpolation for Time Series Forecasting[J]. References [9]: WOO G, LIU C, SAHOO D, et al. ETSformer: Exponential Smoothing Transformers for Time-series Forecasting[J]. References

[10] : NIE Y, NGUYEN NamH, SINTHONG P, et al. A TimeSeries is Worth 64Words:Long-term Forecasting with Transformers[J].2022.) shows satisfactory results in time series forecasting.

[0004] Therefore, in order to enable neural networks to learn the potential relationships between multiple time points in a time series and to identify anomalies in the time series based on the extracted features, this project proposes an unsupervised time series anomaly detection method based on multi-scale reconstruction, combining time series reconstruction and time series decoupling. The method converts the time series into multi-scale time data, which is then input into the neural network for anomaly detection. This improves the anomaly detection rate while effectively combining time series reconstruction and time series decoupling. Summary of the Invention

[0005] To overcome the shortcomings of existing deep learning-based unsupervised multivariate time series anomaly detection methods that rarely utilize multi-scale information and reconstruct neural networks to improve detection accuracy, this invention provides an unsupervised time series anomaly detection method based on multi-scale reconstruction. This method utilizes the multi-scale information of multivariate time series to fully capture the potential features between them. By decoupling at multiple scales and deeply capturing and reconstructing residual information, it not only improves the accuracy of anomaly detection but also greatly enhances the robustness of the model.

[0006] The technical solution adopted by this invention to solve its technical problem is as follows:

[0007] An unsupervised time series anomaly detection method based on multi-scale reconstruction includes the following steps:

[0008] S1 uses a time series anomaly detection dataset, which is divided into training set, validation set and test set according to a set ratio. The training set is used to train the model, the validation set is used to assist the model training, and the test set is used to test the anomaly detection accuracy of the trained model. The multi-scale time decoupling module uses flat decoupling to process the length of multivariate time series, generates multiple time series data sets of different scales, and provides corresponding masking methods.

[0009] S2, input the multiple sets of data that have completed multi-scale decoupling into the multiple residual reconstruction network generator models we have constructed for parallel training, obtain multiple reconstructed data, calculate the reconstruction residual value and joint weight through the reconstruction residual collaborative learning module, and integrate multiple pseudo-reconstructed time series into a reconstructed time series through the joint weight;

[0010] S3: The original time series is captured by Fourier transform to obtain the frequency, periodicity and trend information of the time series. At the same time, perturbation and detail loss processing are added to its low-frequency part to generate a pseudo-anomaly time series. This pseudo-anomaly time series and the reconstructed time series generated in step 2 are input into the discriminator to obtain the adversarial training loss. Meanwhile, the reconstruction loss of the reconstructed time series and the original time series are calculated. Finally, the model parameters are updated based on the adversarial training loss and the reconstruction loss to complete the model training.

[0011] S4 involves inputting the test set data into the trained model for reconstruction. The anomaly score is determined by the cosine similarity between the reconstructed time series and the original time series, thus completing the anomaly detection for the time series.

[0012] The beneficial effects of this invention are as follows: by decoupling the time series, features at different scales of the time series are obtained as input, improving model performance; multiple residual reconstruction neural networks are used to reconstruct the input time series, capturing latent features of time series at different scales, further improving model performance; through a reconstructed residual collaborative learning module, the extracted latent features are selectively combined, improving model performance; and synthetic pseudo-anomaly samples are added to the model's discriminator, increasing the model's generalization ability. This invention can improve detection accuracy and increase the robustness of the model. Attached Figure Description

[0013] Figure 1 This is a flowchart of the unsupervised time series anomaly detection method based on multi-scale reconstruction proposed in this invention.

[0014] Figure 2This is a framework diagram of the unsupervised time series anomaly detection method based on multi-scale reconstruction proposed in this invention.

[0015] Figure 3 This invention provides a time series multi-scale decoupling module.

[0016] Figure 4 This invention relates to the time series reconstruction residual collaborative learning module.

[0017] Figure 5 This invention provides a pseudo-anomaly time series generation module. Detailed Implementation

[0018] The invention will now be described in further detail below with reference to the accompanying drawings, using an example of an abnormal CAN data set in a vehicle network. (Refer to...) Figures 1-5 An unsupervised time series anomaly detection method based on multi-scale reconstruction includes the following steps:

[0019] S1: A time-series anomaly detection dataset is used, divided into training, validation, and test sets according to a certain ratio. The training set is used to train the model, the validation set is used to assist model training, and the test set is used to test the anomaly detection accuracy of the trained model. Multivariate time series are processed through a flattening decoupling method in the multi-scale time decoupling module, generating multiple time-series data sets at different scales, and providing corresponding masking methods (see [reference]). Figure 2 and Figure 3 :

[0020] In this example, the CAN anomaly dataset of the vehicle network is selected as the dataset for this invention. The normal dataset without anomalies is used as the training set and validation set, divided sequentially in a 5:1 ratio. The dataset with anomalies is used as the test set to test the detection accuracy of the model. The data in the training set is defined as X:

[0021] X = (x1, x2, ... x) T (1)

[0022] in C represents the feature dimension of the time series, T represents the length of the time series, and x represents the feature dimension of the time series. i This represents the feature vector at the i-th time point of the time series. The multivariate time series X is input into the multiscale decoupling module for time series decoupling, resulting in multiple time series at different scales.

[0023]

[0024] in P represents the level of scale used to divide the time series. A larger P value indicates a more granular division of the original time series, with a smaller scale. The time series length is also shorter. At the same time, since some intra-scale information is lost due to segmentation at different scales, the segmentation method follows the segmentation relationship between adjacent scales to reduce the loss of intra-scale information:

[0025]

[0026] The obtained time series of the same scale They are stacked sequentially into a three-dimensional tensor V. P :

[0027]

[0028] in To enable the model to extract latent features from the time series data, an alternating masking method is used to remove specific portions of the data, resulting in the input tensor Z. P :

[0029]

[0030] in This indicates that in a three-dimensional tensor The elements located in dimensions k, i, and j. Represents a three-dimensional tensor Elements located at dimensions k, i, and j, satisfying the following conditions: Masking operations do not change the dimensions, therefore In other words, based on the above operations, L tensors Z representing different time scales will be obtained. P P∈(0,L], repeat the above operation on the validation set to obtain the final tensor Z′. P ,P∈(0,L).

[0031] S2: Input the multiple sets of data that have completed multi-scale decoupling into the multiple residual reconstruction network generator models we have constructed for parallel training to obtain multiple sets of reconstructed data. Calculate the reconstructed residual values ​​and joint weights through the reconstruction residual collaborative learning module, and integrate the multiple pseudo-reconstructed time series into a reconstructed time series through the joint weights.

[0032] Reference Figure 2 and Figure 4To better extract latent features from time series data for anomaly detection, an adversarial generative network (GAN) is used for feature extraction. The network generator is defined as G, and the network discriminator as D. The generator structure uses a convolution with a stride of 4 to downsample the input data to 1 / 4 of its size, uses 8 residual blocks to extract features, and then upsamples the features to the size of the input data. Since the differences between normal and anomalous time points vary across different scales, L generators {G1, G2, ..., G...} are used. L}, for tensors Z at different scales P Perform parallel feature extraction:

[0033]

[0034] Among them, the pseudo-reconstruction tensor Network Generator G P ,P∈(0,L). Since features at different scales are complementary, to enhance the interaction between features at different scales, multiple sets of pseudo-reconstruction tensors are used to calculate the reconstruction residual value through a reconstruction residual collaborative learning module, as shown in the following formula:

[0035]

[0036] in Using (1-M) P The dot product is used to assess the effectiveness of the model's reconstruction of the mask portion. After each round of model training, multiple reconstruction residuals are obtained. To simultaneously capture both global and local features of the reconstruction residuals, global max pooling and global average pooling are employed.

[0037]

[0038] in Let Rs(Z) represent the reconstructed residuals calculated in the h-th training round, based on global maximization and average pooling, respectively. P ) i,j Let represent the elements of the i-th and j-th dimensions. A validation set is used to assist training and balance the feature extraction performance of multiple generators. The above operations are repeated on the validation set data, and the mean squared error between the validation set and the training set is calculated.

[0039]

[0040] Where Z′ P The tensor obtained from the validation set through the multi-scale decoupling module exhibits a large generalization error. This means that the model generator G PThe generalization ability is poor in the h-th round, so less attention should be paid to the generator G during updates. P Therefore, the joint weights are defined as the reciprocal of their generalization error:

[0041]

[0042] in This represents the joint weights of the P-th model generator trained in the h-th round, and also relates to the obtained pseudo-reconstruction tensor. Perform a dimensionality transformation to match the original time series. Then multiply it by the joint weights to obtain the final reconstructed time series.

[0043]

[0044] in This represents the reconstructed time series obtained in the h-th training round.

[0045] S3 involves using a Fourier transform to capture information such as frequency, period, and trend from the original time series. Simultaneously, perturbations and detail loss processing are added to the low-frequency components to generate pseudo-anomaly time series. These, along with the reconstructed time series generated in step 2, are input into the discriminator to obtain the adversarial training loss. Simultaneously, the reconstruction losses of the reconstructed and original time series are calculated. Finally, the model parameters are updated based on the adversarial training loss and the reconstruction loss to complete model training.

[0046] Reference Figure 5 Since the model was never trained on anomalous data, data augmentation was used to generate fake anomalous samples to aid training. Specifically, for the normal data used in the training phase, perturbations and missing details were applied to each normal data point to generate fake anomalous data. First, Gaussian blur was used to achieve the missing details operation, with the Gaussian kernel h having elements h. i The values ​​are obtained from a Gaussian distribution:

[0047]

[0048] Among them, Gaussian convolution kernel σ controls the range of missing details; in this example, σ = 0.5. The pseudo-anomaly sample X is obtained using a Gaussian convolution kernel. blur :

[0049] X blur =X*h (14)

[0050] Where * denotes a convolution operation. To increase the variety of pseudo-anomaly samples and enhance the model's generalization ability, pseudo-anomaly samples are generated by adding perturbations to the low frequencies of the data. First, a Fourier transform is used to capture the low-frequency components of the time series:

[0051] X f =F(X) (15)

[0052] Where F represents the Fourier transform, X f This is a frequency domain representation, where the low-frequency range is f∈(0,f). c The low-frequency cutoff frequency is f. c A perturbation is added to the low-frequency range of the data, and the perturbation anomaly sample X is obtained through inverse Fourier transform. per :

[0053] X per =F -1 (X f +ε|f∈(0,f c (16)

[0054] Where F -1 The inverse Fourier transform is used to input the obtained pseudo-anomaly samples and reconstructed data into the discriminator to calculate the adversarial training loss L. adv :

[0055]

[0056] Where X A This is represented as a pseudo-anomaly sample X. per ,X blur The discriminator D consists of four convolutional layers with a kernel size of 3 and one linear layer. To ensure that the data reconstructed by the generator network closely resembles the original data, cosine similarity is used as the reconstruction loss L. CON :

[0057]

[0058] The total loss L is calculated using the two losses mentioned above. all Update the model:

[0059] L all =L CON +ζL adv (19)

[0060] Where ζ is the ratio balancing the two losses, in this example ζ = 0.2. Multiple model weights are updated using this loss function to complete model training.

[0061]

[0062] The model's weight set is {w1, w2, ..., w L}, each w i Let η represent the weight vector of the i-th model, and η be the learning rate.

[0063] S4. Input the test set data into the trained model for reconstruction. Determine the anomaly score by the cosine similarity between the reconstructed time series and the original time series, thus completing the anomaly detection for the time series.

[0064] Reference Figure 1 and Figure 2 The test set data (containing anomalous samples) is input into the trained model to calculate the cosine similarity, which is then used as the anomalous score.

[0065]

[0066] Where x i This represents the data at the i-th time point in the time series data. Based on the anomaly score calculated point by point, a threshold δ is used to determine whether a point is an anomaly. If the score is less than the threshold, the output y is considered an anomaly. That is:

[0067]

[0068] The abnormal threshold δ is set based on an abnormal data rate of 0.5% in the CAN test set.

[0069] The above describes an embodiment of the unsupervised anomaly detection method using multi-scale reconstruction in the context of using anomaly datasets from CAN (Automotive Network) systems. This invention enhances the time series features by processing the original time series through multi-scale decoupling and masking; it trains the residual reconstruction network using the processed time series and balances the reconstruction performance of multiple models through a collaborative learning module for the reconstruction residuals, resulting in better time series reconstruction; it generates pseudo-anomaly time series using perturbation and missing details, allowing the generator to better complete sequence reconstruction; and it significantly increases the robustness of the model while improving the accuracy of anomaly time series detection. This invention is merely illustrative and not restrictive. Those skilled in the art will understand that many changes, modifications, and even equivalents can be made within the spirit and scope defined by the claims, all of which will fall within the protection scope of this invention.

Claims

1. An unsupervised time series anomaly detection method based on multi-scale reconstruction, characterized in that: The following steps are involved: S1, uses a variety of public time series anomaly detection data sets, divides them into training set, validation set and test set according to a certain ratio, the training set is used to train the model, the validation set is used to assist model training, and the test set is used to test the anomaly detection accuracy of the trained model. The multivariate time series length is processed by multi-scale decoupling, multiple time series matrix sets of different scales are generated, and corresponding masking methods are provided; S2, input multiple sets of data that have completed multi-scale decoupling into multiple residual reconstruction network generator models we constructed for parallel training, obtain multiple sets of reconstructed data, calculate the reconstructed residual values ​​and joint weights through the reconstruction residual collaborative learning module, and integrate multiple pseudo-reconstructed time series into a reconstructed time series through the joint weights; S3, the original time series is captured by Fourier transform in the frequency, period and trend information of the time series, and the low-frequency part is disturbed and the detail loss is processed to generate a pseudo abnormal time series, and it and the reconstructed time series generated in step S2 are input into the discriminator to obtain the adversarial training loss; at the same time, the reconstruction loss of the reconstructed time series and the original time series is calculated; finally, the model parameters are updated based on the adversarial training loss and the reconstruction loss to complete the model training; S4, input the data of the test set into the trained model for reconstruction, determine the anomaly score through the cosine similarity between the reconstructed time series and the original time series, and complete the anomaly detection for the time series.

2. The unsupervised time series anomaly detection method based on multi-scale reconstruction according to claim 1 is characterized in that: Step S1 includes the following sub-steps: S1.1: Define the data of the training set as X: X=(x1,x2,...x T ) (1) in C is the characteristic dimension of the time series, T is the length of the time series, x i Represents the feature vector of the i-th time point in the time series. The multivariate time series X is input into the multi-scale decoupling module for time series decoupling to obtain multiple time series of different scales. in P represents the level of time series scale division. The larger the P value, the more detailed the division of the original time series and the smaller the scale. The time series length is also shorter; at the same time, since segmentation at different scales will lose some information within the scale, the division method follows the segmentation relationship between adjacent scales to reduce the loss of information within the scale: The obtained time series of the same scale Stacked sequentially into a three-dimensional tensor V P : in In order to enable the model to extract the potential features in the time series, an alternating mask method is used to delete specific parts of the data to obtain the input tensor Z P : in Then it means that in the three-dimensional tensor The elements in the k-, i-, and j-dimensions of Represented in a three-dimensional tensor The elements in the k, i, and j dimensions satisfy Masking does not change the dimensions, so Get L tensors Z representing different time scales P ,P∈(0,L], repeat the above operation and apply it to the validation set to obtain the final tensor Z′ P ,P∈(0,L].

3. The unsupervised time series anomaly detection method based on multi-scale reconstruction according to claim 2 is characterized in that: The process of step S2 is as follows: Data feature extraction is performed based on the adversarial generative network. The network generator is defined as G, the network discriminator is defined as D, and the structure of the generator uses a convolution with a step size of 4 to downsample the input data to 1 / 4 of the input size. 8 residual blocks are used to extract features and upsample the features to the size of the input data. L generators G are used. P ,P∈(0,L] for tensors Z of different scales P Perform parallel feature extraction: The pseudo-reconstructed tensor Since features of different scales are complementary, in order to strengthen the interaction of features of different scales, multiple sets of pseudo-reconstructed tensors are passed through the reconstruction residual collaborative learning module to calculate the reconstruction residual value. The formula is as follows: in Using (1-M P ) dot multiplication is to focus on whether the model is effective in reconstructing the masked part. After completing each round of model training, multiple reconstructed residual values ​​will be obtained. In order to capture the global and local features of the reconstructed residual values ​​at the same time, global maximum pooling and global average pooling are used: in They represent the reconstructed residual values ​​of global maximization and average pooling calculated in the h-th round of training, Rs(Z P ) i,j Represents the elements of the i-th dimension and the j-th dimension. The validation set is used to assist the training, balance the feature extraction effects of multiple generators, repeat the above operation on the validation set data, and calculate the mean square error with the training set: where Z′ P Represents the tensor obtained by the validation set through the multi-scale decoupling module, and the joint weight is defined as the inverse of its generalization error: in represents the joint weight of the Pth model generator in the hth round of training, and the pseudo-reconstructed tensor obtained Transform the dimension to be consistent with the original time series. And multiply it with the joint weight to get the final reconstructed time series in represents the reconstructed time series obtained in the hth training round.

4. The unsupervised time series anomaly detection method based on multi-scale reconstruction according to claim 1 or 3, characterized in that: The process of step S3 is as follows: For the normal data used in the training phase, perturbations and detail loss are applied to each normal data to generate fake abnormal data. First, Gaussian blur is used to achieve the operation of data detail loss. The element h of the Gaussian kernel h is i are the values ​​obtained from a Gaussian distribution: The Gaussian convolution kernel σ controls the range of missing details, and the pseudo-abnormal sample X is obtained through the Gaussian convolution kernel. blur : X blur =X*h (14) Where * represents the convolution operation. At the same time, in order to increase the types of pseudo-abnormal samples and enhance the generalization of the model, pseudo-abnormal samples are generated by adding disturbances to the low frequency of the data. First, the low-frequency part of the time series is captured by Fourier transform: X f =F(X) (15) Where F represents Fourier transform, X f It is a frequency domain representation, assuming that the low frequency range is f∈(0,f c ), the low frequency cutoff frequency is f c , add disturbance in the low-frequency range of the data, and obtain the disturbance abnormal sample X through inverse Fourier transform per : X per =F -1 (X f +ε|f∈(0,f c )) (16) where F -1 Represents the inverse Fourier transform, and the obtained pseudo-abnormal samples and reconstructed data are input into the discriminator to calculate the adversarial training loss L adv : Where X A Represented as a pseudo-abnormal sample X per ,X blur The discriminator D consists of four convolutional layers with a convolution kernel of 3 and one linear layer. At the same time, in order to ensure that the data reconstructed by the generator network is closer to the original data, the cosine similarity is used as the reconstruction loss L CON : The above two losses are used as the total loss L all To update the model: L all =L CON +ζL adv (19) Among them, ζ is the ratio of balancing the two losses, and multiple model weights are updated through this loss function to complete the model training: The weight set of the model is {w1,w2,...,w L }, each w i represents the weight vector of the i-th model, and η is the learning rate.

5. The unsupervised time series anomaly detection method based on multi-scale reconstruction according to claim 1 or 2, characterized in that: The process of step S4 is as follows: Input the test set data with abnormal samples into the trained model to calculate the cosine similarity and use it as the anomaly score: where x i Represents the i-th time point data in the time series data, According to the anomaly score calculated by point, a threshold δ is used to determine whether a point is an anomaly. If the score is less than the threshold, the output y is an anomaly, that is:

Images