Unsupervised anomaly detection method for time series data

Abstract

The invention discloses an unsupervised anomaly detection method for time series data, and relates to the technical field of anomaly detection of the time series data, which comprises the following steps of: filling a current null value by using an average value of a forward value and a backward value of the current null value data, and performing data normalization processing on numeric data in a data set. The method comprises the following steps: normalizing data to standard normal distribution with a mean value of 0 and a standard deviation of 1, and separating a training set and a test set from a given time sequence data set; according to the unsupervised anomaly detection method for the time series data, potential representation is performed on the data through a beta-VAE encoder, a long-term and short-term memory network module is adopted to estimate the long-term correlation of the potential representation, the forward and backward dependency relationship of the data is captured, and time series prediction is performed on the data. And finally, through average fusion of prediction results and reconstruction of a beta-VAE decoder, a reconstruction error is calculated to detect time sequence abnormity. The method has excellent anomaly detection performance on a plurality of public time series data sets.

Description

Technical Field

[0001] The present invention relates to the technical field of time series data anomaly detection, and in particular to an unsupervised anomaly detection method for time series data. Background Art

[0002] Time series data is widely used in many fields such as financial markets, medical monitoring, industrial process control, and geological disaster warning. Effective time series anomaly detection can help identify potential problems and predict system failures, thereby improving system reliability and security. However, time series data is usually complex and diverse, which brings many challenges to anomaly detection.

[0003] Faced with challenges, researchers have explored anomaly detection methods from different perspectives. Among them, unsupervised anomaly detection methods do not rely on labeled data, and detect anomalies by learning the normal patterns of data. Compared with supervised learning methods, unsupervised methods are more general and are particularly suitable for scenarios where it is difficult to obtain sufficient labeled data in practical applications. Common unsupervised anomaly detection methods include methods based on statistics, machine learning, and deep learning. However, traditional anomaly detection methods have many limitations when processing complex time series data. For example, statistical methods rely on the statistical characteristics of data, and their main disadvantages are strong assumptions, complex parameter selection, limited ability to handle complex and nonlinear patterns, and poor ability to cope with long-term dependencies and data pattern changes. Machine learning-based methods have a low degree of automation and require manual feature engineering and parameter tuning.

[0004] With the vigorous development of computer artificial intelligence, deep learning methods have gradually shown their advantages in anomaly detection, but there are also some shortcomings. For example, autoencoders (AE) identify anomalies by compressing and reconstructing data, but it is usually difficult to capture the temporal dependencies in time series. Recurrent Neural Networks (RNN), especially Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), are good at processing sequence data, but they often require large-scale labeled data for training in anomaly detection. Variational Auto-Encoders (VAE) reconstruct and generate by learning the probability distribution of data, but standard VAE has the problem of unclear latent space structure when processing complex time series.

[0005] In order to overcome the shortcomings of existing methods, this paper proposes an unsupervised time series anomaly detection method that combines β-VAE and BiLSTM. The advantages of the two are combined to design a complete time series anomaly detection framework. First, the β-VAE module encoder is used to encode the time series data, and the local information in the window is represented by low-dimensional embedding. The decoupling and expressiveness of the latent space are enhanced by introducing hyperparameters, so as to better capture the intrinsic characteristics of the data. Then, the BiLSTM model is used to process the low-dimensional embedding generated by the β-VAE module encoder, manage long-term sequence patterns and predict the latent representation. The forward and backward information of the bidirectional LSTM is fully utilized by averaging the fusion prediction results to improve the modeling ability of time series dependencies. Finally, the prediction results of BiLSTM are input into the β-VAE module decoder for reconstruction, and anomaly detection is performed by calculating the reconstruction error, effectively identifying anomalies and potential anomaly areas. This method provides a novel and effective solution for time series anomaly detection. Summary of the invention

[0006] To achieve the above objectives, the present invention is implemented by the following technical solution: an unsupervised anomaly detection method for time series data, comprising the following steps:

[0007] Step 1: Preprocess the acquired time series data. The specific steps are as follows:

[0008] S1, fill the empty values ​​in the current data set with the average of the forward value and the backward value of the current data;

[0009] S2. Normalize the numerical data in the data set to a standard normal distribution with a mean of 0 and a standard deviation of 1.

[0010] S3. Separate the training set and the test set from the given time series data set, select the continuous time series without anomalies in the time series as the training data, and the remaining time series containing anomalies as the test data for model evaluation;

[0011] Then extract 10% of the data from the training set as the validation set, which is completely separate from the training set and used for model verification and debugging;

[0012] S4, slicing and expanding the data set to be input into the β-VAE module encoder for data feature extraction to form a β-VAE module training set;

[0013] The training set uses the overlapping window method to generate multiple sliding windows to increase the number of windows to be trained. Specifically, given the number of original training sets is n train , the sliding window length is l, then the number of sliding windows generated is

[0014] n win =n train -l+1

[0015] S5, re-slicing and expanding the original training set to form a BiLSTM module training set for training the BiLSTM module to perform time-dependent prediction of potential representations;

[0016] The sliding window and non-overlapping window methods are used on the training set of the BiLSTM model to generate multiple training sequences from the time series data. The specific steps are as follows:

[0017] a1. Generate non-overlapping windows: First, generate non-overlapping windows of fixed length according to the sliding window size and the number of training samples. The number of non-overlapping windows is:

[0018]

[0019] Where k represents the starting offset of the sliding window, k∈[1,…,l];

[0020] a2. Generate BiLSTM input sequence; then generate BiLSTM input sequence. For each starting offset k, the number of generated BiLSTM sequences is:

[0021] n cur =n not -s+1

[0022] Where s is the number of sliding windows in each sequence;

[0023] a3. Combine all sequences. Combine all generated BiLSTM sequences into a complete training sequence set, the number of which is:

[0024]

[0025] Right now:

[0026]

[0027] Among them, the tth sequence is:

[0028] W t =[w t ,…,w t+(s-2)×l ,w t+(s-1)×l ].

[0029] Step 2: Build an unsupervised time series data anomaly detection framework. This anomaly detection framework is based on VAE. The focus is on introducing Bi-LSTM modules in the encoder module and decoder module to consider the bidirectional dependency of time series data and fuse the bidirectional output results.

[0030] Step 3: Train the β-VAE module and the BiLSTM module;

[0031] Step 4: For the reconstructed window, define a scoring function to mark the abnormal behavior of the window, and accumulate the prediction error of each window;

[0032] Detect anomalies, define a threshold on the scoring function, and use the grid search method to find the optimal threshold. If the threshold is exceeded, an abnormal alarm will be triggered at the current moment. The corresponding sequence will be marked as a suspicious sequence, which may have an anomaly and marked as an abnormal area;

[0033] Step 5: Input the test set into the trained time series anomaly detection model to analyze and evaluate the time series anomaly;

[0034] The unsupervised time series data anomaly detection framework in step 2 includes:

[0035] The encoder structure takes a local window of l consecutive samples as input and receives input data of shape (b, l, c), where b is the batch size, l is the window length, and c is the number of channels;

[0036] After the input data enters the encoder, it is processed by a series of convolutional layers. The specific encoder structure includes: three sliding window lengths of l=24, l=48 and l=144 are designed as input, the convolution kernel size is 3×c, and then three layers of convolutional layers are connected in series for deep feature extraction. Then different convolution kernels are used for different sliding windows to extract different levels of features. Finally, a full connection operation is performed to map the mean μ parameter and standard deviation σ parameter of the latent space in the form of low-dimensional embedding;

[0037] The mean μ parameter and the standard deviation σ parameter are used to define the distribution of the latent variable, and the encoded feature vector is generated by sampling the multivariate normal distribution;

[0038] Bidirectional long short-term memory module, including forward long short-term memory network module and reverse long short-term memory network module, averages and fuses the results obtained in the two directions during the implementation stage;

[0039] The decoder structure receives the encoded feature vector or randomly sampled latent vector as input and reconstructs the original signal;

[0040] According to the different lengths of the input window, the decoder structure will be adjusted accordingly, using inverse convolution and transposed convolution operations to gradually reconstruct the window. The final output is the reconstructed (b, l, c) shape signal data;

[0041] The decoder structure is to adjust the data structure of the input data through a fully connected layer and a reshap layer, then pass through a convolution layer and use a 1×c convolution kernel and a reshap layer to adjust the data structure again, then the convolution kernel uses three convolution layers of size 3×c and a reshape layer to deeply restore the data features and adjust the data structure, and finally use 9×c and 5×c convolution kernels and a reshape layer to reconstruct the data and restore it to three sliding window lengths of l=24, l=48 and l=144 that are the same as the input data.

[0042] The specific steps for training the β-VAE module and the BiLSTM module in step 3 are as follows:

[0043] b1. Input the β-VAE module training set processed in step 1 into the β-VAE module to train the β-VAE module. The improved ELBO loss is used as the loss function during the training process;

[0044] b2. The BiLSTM module training set processed in step 1 is encoded by the β-VAE encoder module and input into the BiLSTM module to train the module;

[0045] The parameters of the neural network are trained after setting the optimizer, loss function, learning rate, and batchsize. The model uses mean square error as the loss function and uses the Adam optimizer for parameter optimization. During the training process, the model is batch trained using pre-generated training data and validation data. The batch size, number of training rounds, and callback function are set to monitor and optimize the training process;

[0046] b3. Average the results and input them into the β-VAE decoder for reconstruction; calculate the reconstruction error between the reconstructed data and the original data.

[0047] The present invention provides an unsupervised anomaly detection method for time series data. It has the following beneficial effects:

[0048] First, this unsupervised anomaly detection method for time series data uses a long short-term memory network module to estimate the long-term correlation of the potential representation, capture its forward and backward dependencies, and perform time series prediction on it. Finally, the prediction results are averaged and fused, and the decoder is reconstructed to calculate the reconstruction error to detect time series anomalies. This method has superior anomaly detection performance on multiple public time series data sets, can effectively extract time-related complex data features, achieve efficient calculation and real-time anomaly detection, identify abnormal points and abnormal areas, improve detection accuracy and robustness, and reduce false positives and false negatives.

[0049] Second, this unsupervised anomaly detection method for time series data, the advantages of the β-VAE-BiLSTM model in disentangled representation learning, bidirectional dependency capture, reconstruction error sensitivity and model interpretability make it beneficial for time series anomaly detection.

[0050] 3. This unsupervised anomaly detection method for time series data enhances the disentangled representation ability of the latent space by introducing the hyperparameter β in the ELBO objective function of β-VAE, so that each latent variable independently controls a specific factor in the time series. This disentangled representation is crucial to capturing the essential characteristics of time series data, which can be fully utilized in the time-dependent modeling process of BiLSTM. Through the latent representation obtained by the β-VAE encoder, the time series data is converted into a structured and more interpretable representation, making the subsequent BiLSTM prediction more accurate and efficient.

[0051] Fourth, this unsupervised anomaly detection method for time series data performs well in capturing bidirectional dependencies in time series through BiLSTM. Anomalies in time series data often depend on past and future data points. Through bidirectional LSTM units, BiLSTM can more comprehensively capture the contextual information of time series data. Combined with the potential representation of β-VAE, BiLSTM is able to use richer features for modeling, thereby improving the accuracy of anomaly detection. This fusion of bidirectional information allows the model to utilize complete contextual information at each time point, enhancing the ability to recognize complex abnormal patterns.

[0052] 5. This unsupervised anomaly detection method for time series data first preprocesses the data set, especially designs an expansion method for the data, so that it still has strong anomaly detection capabilities on data sets with small sample sizes. On the basis of retaining the overall framework of the VAE model encoder and decoder, the internal network structure of the encoder and decoder is redesigned, and the ELBO loss of the VAE model is further optimized to further improve its representation learning ability. In order to enhance the time dependency of the VAE model, a bidirectional LSTM structure with long-term dependency is added between the encoder and decoder, and the prediction coding results of the bidirectional LSTM are averaged and fused. Finally, the anomaly detection framework of β-VAE encoder-BiLSTM-β-VAE decoder is designed. The most suitable reconstruction error threshold is found through the grid search method to achieve the goals of offline detection of existing data anomalies and anomaly ranges and real-time detection of data anomalies and anomaly areas.

[0053] 6. This unsupervised anomaly detection method for time series data, the β-VAE-BiLSTM method, due to its powerful modeling ability and ability to capture complex dependencies, can adapt to different types of time series data and perform well in various application scenarios contained in the tested dataset. Whether it is anomaly detection for real-time monitoring or anomaly mining for historical data analysis, this method can provide efficient and accurate solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of a training process of an unsupervised anomaly detection method for time series data according to the present invention;

[0055] Figure 2 This is a schematic diagram of the abnormality detection process of the present invention; DETAILED DESCRIPTION

[0056] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention are provided for the purpose of illustration and description, and are not intended to be exhaustive or to limit the present invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention, and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

[0057] like Figure 1 to Figure 2 As shown, the present invention provides a technical solution: an unsupervised anomaly detection method for time series data, comprising the following steps:

[0058] Step 1: Preprocess the time series data. In this example, the internal temperature time series data set of an industrial machine is processed. The processing operations include: filling in null values, normalizing the data, separating the training set from the test set, and expanding the data slices.

[0059] Check whether there are null values ​​in the data set, and fill the null values ​​in the current data set with the average of the null forward and backward values.

[0060] Standardize the numerical data in the dataset.

[0061] Separate the training set and test set from the given time series data set, select the continuous time series without anomalies in the time series as the training data, and the remaining time series containing anomalies as the test data for model evaluation. Then extract 10% of the data in the training set as the validation set, which is completely separate from the training set for model verification and debugging. This data contains a total of 22,695 data, and select the middle 5000 to 15,000 data as the training set. The remaining data from 1 to 5000 and from 15,000 to 22,695 are used as the test set.

[0062] Perform data slicing and expansion operations on the training set. The specific operations are as follows:

[0063] 4.1 Set the sliding window length, which is set to l=48 for this dataset.

[0064] 4.2 Use a sliding window to move the window with a step size of 1, divide the data set into a training window set of the β-VAE module containing (10000-48+1=9953) windows, and input the β-VAE module for training.

[0065] The original data set is re-sliced ​​and expanded to form a BiLSTM module training set, which is used to train the BiLSTM module for time-dependent prediction. The sliding window and non-overlapping window methods are used on the training set of the BiLSTM model to generate multiple training sequences from the time series data. The specific steps are as follows:

[0066] 5.1 Generate non-overlapping windows. First, generate non-overlapping windows of fixed length according to the sliding window size and the number of training samples. The number of non-overlapping windows is

[0067]

[0068] Where k represents the starting offset of the sliding window, k∈[1,…,48].

[0069] 5.2 Generate BiLSTM input sequence. Then generate the input sequence of BiLSTM. For each starting offset k, the number of generated BiLSTM sequences is

[0070] n cur =472-12+1=461

[0071] Among them, s is the number of sliding windows in each sequence, which is set to 12 for this dataset.

[0072] 5.3 Combine all sequences. Combine all generated BiLSTM sequences into a complete training sequence set, the number of which is

[0073]

[0074] Step 2: Build an unsupervised time series data anomaly detection framework. This model is based on VAE. The focus is on introducing Bi-LSTM modules in the encoder module and decoder module to consider the bidirectional dependency of time series data and fuse the bidirectional output results. The specific structure is as follows:

[0075] The first part is the encoder structure.

[0076] 1.1 In this dataset, the encoder takes a local window containing l = 48 consecutive samples in batches as input and receives input data of shape (32, 48, 1), where 32 is the batch size, 48 is the window length, and 1 is the number of channels.

[0077] 1.2 Figure 1 As shown in the figure, after the input data enters the encoder, it is processed by a series of convolutional layers. The specific encoder structure includes: the window length l = 48 is selected as the input, the convolution kernel size is 3×1, and then three layers of convolutional layers are connected in series for deep feature extraction. Then different convolution kernels are used for different sliding windows to extract different levels of features. Finally, a full connection operation is performed to map the mean μ parameter and standard deviation σ parameter of the latent space in the form of low-dimensional embedding. These parameters are used to define the distribution of the latent variables, and the encoded feature vector is generated by sampling the multivariate normal distribution.

[0078] The second part is a bidirectional long short-term memory module, which includes a forward long short-term memory network module and a reverse long short-term memory network module. In the implementation stage, the results obtained in the two directions need to be averaged and fused.

[0079] The third part is the decoder structure.

[0080] 3.1 The decoder receives the encoded feature vector or randomly sampled latent vector as input and reconstructs the original signal. Depending on the length of the input window, the decoder structure will be adjusted accordingly, using inverse convolution and transposed convolution operations to gradually reconstruct the window. The final output is the reconstructed signal data of (32,48,1) shape.

[0081] 3.2 The main function of the encoder structure is to reconstruct the low-dimensional potential representation into a standard data form. The specific decoder structure is to adjust the data structure of the input data through the full connection layer and the reshap layer, and then pass through the convolution layer and use the 1×1 convolution kernel and the reshap layer to adjust the data structure again. After that, the convolution kernel uses a three-layer convolution layer of 3×1 size and a reshape layer to deeply restore the data features and adjust the data structure. Finally, 9×1 and 5×1 convolution kernels and reshape layers are used to reconstruct the data and restore it to a sliding window length of l=48.

[0082] Step 3: Train the β-VAE module and the BiLSTM module.

[0083] The β-VAE encoder module training set processed in step 1 is input into the β-VAE module to train the β-VAE module. The loss function uses the improved ELBO loss function Among them, β=2 and the learning rate is set to 0.0004.

[0084] The BiLSTM module training set processed in step 1 is encoded by the β-VAE encoder module and input into the BiLSTM module for training. The optimizer is set to Adam optimizer, the mean square error is used as the loss function, the learning rate is set to 0.0002, the batch size is set to 32, and the number of iterations is set to 20. Some parameters can be slightly adjusted in the actual data set application. During the training process, the model uses the pre-generated training data and validation data for batch training. The batch size, number of training rounds, and callback function are set to monitor and optimize the training process.

[0085] The results are averaged and fused, and then input into the β-VAE decoder for reconstruction. The reconstruction error between the reconstructed data and the original data is calculated.

[0086] Step 4: Define the scoring function d for the reconstructed window t To mark abnormal behavior of the window, accumulate W t The prediction error of each window in . To detect anomalies, continue in the scoring function d t A threshold θ is defined on the grid, and the optimal threshold is found through grid search. If the threshold is exceeded, an abnormal alarm y will be triggered at the current time t. t = 1. The corresponding sequence W t It will be marked as a suspicious sequence, which may have abnormalities and marked as abnormal regions.

[0087] Step 5: Input the test set into the trained time series anomaly detection model to analyze and evaluate the time series anomalies.

[0088] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field and related fields without creative work should fall within the scope of protection of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention are implemented according to the conventional means in the field unless otherwise specified and limited.

Claims

1. An unsupervised anomaly detection method for time series data, characterized in that: The following steps are involved: Step 1: Preprocess the acquired time series data. The specific steps are as follows: S1, fill the empty values ​​in the current data set with the average of the forward value and the backward value of the current data; S2, normalize the numerical data in the data set to a standard normal distribution with a mean of 0 and a standard deviation of 1; S3. Separate the training set and the test set from the given time series data set, select the continuous time series without anomalies in the time series as the training data, and the remaining time series containing anomalies as the test data for model evaluation; Then extract 10% of the data from the training set as the validation set, which is completely separate from the training set and used for model verification and debugging; S4, slicing and expanding the data set to be input into the β-VAE module for data feature extraction to form a β-VAE module training set; The training set uses the overlapping window method to generate multiple sliding windows to increase the number of windows to be trained; specifically, given the number of original training sets is n train , the sliding window length is l, then the number of sliding windows generated is n win =n train -l+1 S5, re-slicing and expanding the original training set to form a BiLSTM module training set for training the BiLSTM module to perform time-dependent prediction of potential representations; The sliding window and non-overlapping window methods are used on the training set of the BiLSTM model to generate multiple training sequences from the time series data. The specific steps are as follows: a1. Generate non-overlapping windows: First, generate non-overlapping windows of fixed length according to the sliding window size and the number of training samples. The number of non-overlapping windows is: Where k represents the starting offset of the sliding window, k∈[1,…,l]; a2. Generate BiLSTM input sequence; then generate BiLSTM input sequence. For each starting offset k, the number of generated BiLSTM sequences is: n cur =n not -s+1 Where s is the number of sliding windows in each sequence; a3. Combine all sequences; combine all generated BiLSTM sequences into a complete training sequence set, the number of which is: Right now: Among them, the tth sequence is: IN t =[in t ,…,In t+(s-2)×l ,In t+(s-1)×l ]; Step 2: Build an unsupervised time series data anomaly detection framework based on VAE; Step 3: Train the β-VAE module and the BiLSTM module; Step 4: For the reconstructed window, define a scoring function to mark the abnormal behavior of the window, and accumulate the prediction error of each window; Detect anomalies. Define a threshold on the scoring function and find the optimal threshold through grid search. If the threshold is exceeded, an abnormal alarm will be triggered at the current moment. The corresponding sequence will be marked as a suspicious sequence, which may have an anomaly and marked as an abnormal area. Step 5: Input the test set into the trained time series anomaly detection model to analyze and evaluate the time series anomalies.

2. The unsupervised anomaly detection method for time series data according to claim 1, characterized in that: The unsupervised time series data anomaly detection framework in step 2 includes: The encoder structure takes a local window of l consecutive samples as input and receives input data of shape (b, l, c), where b is the batch size, l is the window length, and c is the number of channels; After the input data enters the encoder, it is processed by a series of convolutional layers. The specific encoder structure includes: three sliding window lengths of l=24, l=48 and l=144 are designed as input, the convolution kernel size is 3×c, and then three layers of convolutional layers are connected in series for deep feature extraction. Then different convolution kernels are used for different sliding windows to extract different levels of features. Finally, a full connection operation is performed to map the mean μ parameter and standard deviation σ parameter of the latent space in the form of low-dimensional embedding; The mean μ parameter and the standard deviation σ parameter are used to define the distribution of the latent variable, and the encoded feature vector is generated by sampling the multivariate normal distribution; Bidirectional long short-term memory module, including forward long short-term memory network module and reverse long short-term memory network module, averages and fuses the results obtained in the two directions during the implementation stage; The decoder structure receives the encoded feature vector or randomly sampled latent vector as input and reconstructs the original signal; According to the different lengths of the input window, the decoder structure will be adjusted accordingly, using inverse convolution and transposed convolution operations to gradually reconstruct the window, and the final output is the reconstructed (b, l, c) shape signal data; The decoder structure is to adjust the data structure of the input data through a fully connected layer and a reshap layer, then pass through a convolution layer and use a 1×c convolution kernel and a reshap layer to adjust the data structure again, then the convolution kernel uses three convolution layers of size 3×c and a reshape layer to deeply restore the data features and adjust the data structure, and finally use 9×c and 5×c convolution kernels and a reshape layer to reconstruct the data and restore it to three sliding window lengths of l=24, l=48 and l=144 that are the same as the input data.

3. The unsupervised anomaly detection method for time series data according to claim 1, characterized in that: The specific steps for training the β-VAE module and the BiLSTM module in step 3 are as follows: b1. Input the β-VAE module training set processed in step 1 into the β-VAE module to train the β-VAE module. During the training process, the improved ELBO loss is used as the loss function. b2. The BiLSTM module training set processed in step 1 is encoded by the β-VAE encoder module and input into the BiLSTM module to train the module; Set the optimizer, loss function, learning rate, batch size and other parameters, and then train the parameters of the neural network. The model uses mean square error as the loss function and uses Adam optimizer for parameter optimization. During the training process, the model uses pre-generated training data and validation data for batch training, and sets the batch size, number of training rounds, and callback functions to monitor and optimize the training process; b3. Average the results and input them into the β-VAE decoder for reconstruction; calculate the reconstruction error between the reconstructed data and the original data.

Images