Unsupervised time series anomaly detection method based on large language model
Through the combination of the large language model, the embedding layer and attention layer, the detection problem of complex features of multiple time series is solved, and efficient anomaly detection effect is achieved.
Patent Information
- Application Number
- CN202510391216.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-28
- Publication Date
- 2025-07-29
AI Technical Summary
The prior art is difficult to effectively capture the complex features of multiple time series, such as spatial correlation, time dependence and non-stationarity, resulting in poor abnormal detection results.
The large language model is used as the backbone model, combined with the embedding layer and attention layer, time series reconstruction is enhanced through pre-trained language model, and abnormal detection is performed using mean square error and super-threshold peak method.
Accurate anomaly detection of multiple time series is realized, and the fitting ability and detection efficiency of the model are improved.
Smart Images
Figure CN120386980A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data detection, and particularly relates to an unsupervised time series anomaly detection method based on a large language model. Background Art
[0002] A multivariate time series is a complex time series data that has multiple dimensions. It can better capture and describe complex change laws in real life, not limited to a single dimension, and is more suitable for application scenarios with complex change laws in real life. Among them, compared with univariate time series anomaly detection methods, multivariate time series have higher time and space complexity, more invalid and interfering information, and more complex correlation situations between dimensions, which makes the research difficulty of multivariate time series increase exponentially.
[0003] Existing time series anomaly detection methods include: anomaly detection algorithms based on deep learning use deep neural network models to detect anomalies in time series; anomaly detection models based on LSTM use normal data to train the model and use the reconstruction error as the anomaly score; anomaly detection algorithms based on variational autoencoders, which are probabilistic models based on variational inference, use the reconstruction probability to determine anomalies. However, the above methods often cannot capture the overall characteristics / distributions of time series, such as complex spatial correlations and time dependencies, time-varying non-stationarity, and observational incompleteness. Therefore, how to better make the model fit the complexity of time series has become an important topic.
[0004] The latest development of large language models has opened up new possibilities for time series modeling. Large language models such as GPT, GPT-2, LLaMA, etc. have shown strong capabilities in understanding complex dependencies in heterogeneous text data and providing reasonable generations. There is an increasing interest in applying language models to time series tasks, but for large language models with time series, it is still very limited. Summary of the Invention
[0005] In view of the above-mentioned defects of the prior art, the present invention proposes an unsupervised time series anomaly detection method based on a large language model. The technical solution steps designed by the present invention include:
[0006] S1: Preprocess the time series data, set a time window, and divide the time series data into a training set and a test set;
[0007] S2: Extract the time series data features;
[0008] S3: Perform chunking and position embedding processing on the time window;
[0009] S4: Use the large language model as the backbone model and design the embedding layer and the attention layer to construct an anomaly detection model;
[0010] S5: Based on the anomaly detection model, judge whether the time series data is abnormal.
[0011] Preferably, the preprocessing of the time series data in S1 includes:
[0012] Use a Gaussian filter to denoise the time series data. The formula is as follows:
[0013]
[0014] In the formula, y is the denoised time series data, σ is the standard deviation of the Gaussian filter, x[k] is the data point at position k in the time series, k is the position of the current time point, and i is the current time point;
[0015] Normalize the time series data. The formula is as follows:
[0016]
[0017] In the formula, y i is the data point of the processed time series data, x i is the data point at the i-th time point in the time series, x is the average value of x, s is the standard deviation, and n is the number of data points in the time series.
[0018] Preferably, the setting of the time window in S1 includes:
[0019] Convert the data points of the time series data into time windows. The time series data is expressed as X =
[0020] {x1, x2, …, x n}, and represent the local context window with a length of S as W t = x t-S+1 , …, x t , and the window sequence data is F = {W1, W2, …, W t}, where the length of the time window W t is n, and X t ∈R m .
[0021] Preferably, S2 includes:
[0022] Use mean, variance, skewness, and kurtosis statistics as the features of the time series data. The formula is as follows:
[0023]
[0024] Wherein, μ is the mean value, N is the number of samples of the time series data, σ 2 is the variance, sk is the skewness, and p is the kurtosis.
[0025] Preferably, the S3 includes:
[0026] For each input channel x of the time window input i , divide x i into multiple consecutive data blocks, integrate and process each data block, and generate a position encoding vector for the position t of each data block The i-th element of which is defined as follows:
[0027]
[0028] Wherein, pos is the position, d model is the embedding dimension, and i is the dimension index of the embedding vector.
[0029] Preferably, the S4 includes:
[0030] Construct an embedding layer based on the linear probing technique and use an Adapter method, including inserting a module in the Transformer model structure. In the attention layer, the Adapter method introduces an additional parameter matrix to adjust the output of the attention mechanism. The attention layer of the Transformer adopts a cross-attention mechanism, and the formula is as follows:
[0031] Q = XW Q , K = XW K , V = XW V
[0032]
[0033] MultiHead(Q, K, V) = Concat(h1, h2, …, h h )W O , where h i = Attention(Q i , K i , V i )
[0034] Wherein, W Q , W K , W V are learnable parameter matrices, Q, K, and V are vectors derived from the input sequence through preprocessing, and d k is the number of columns of the Q, K matrices.
[0035] Preferably, the S5 includes:
[0036] S501: Train the anomaly detection model using the mean square error of actual values;
[0037] S502: Input the time series data X into the trained anomaly detection model and output the anomaly score;
[0038] S503: Determine the threshold based on the super-threshold peak method;
[0039] S504: Compare the anomaly score with the threshold to detect whether the timestamp is abnormal.
[0040] Preferably, the S501 includes:
[0041] Minimize the difference between the reconstructed value and the actual value. Given the time series data X of length N as the input of the model, denoted as X = {x1, x2, …, x N}, the overall loss function of the anomaly detection model is expressed as follows:
[0042]
[0043] In the formula, Θ is the parameter of the anomaly detection model.
[0044] Preferably, the S502 includes:
[0045] Input the time series data X into the trained anomaly detection model, and for each timestamp x t , calculate the mean variance between it and the reconstructed value as the anomaly score. The formula is as follows:
[0046]
[0047] Preferably, the S504 includes
[0048] For the anomaly score s i of each dimension and the threshold POT(s t ), it is defined by the anomaly diagnosis label. The formula is as follows:
[0049] S i = 1(s i > POT(s t ))
[0050] In the formula, if the time point i is greater than the threshold, then S i is 1, otherwise it is 0.
[0051] Beneficial effects:
[0052] The present invention proposes an unsupervised time series anomaly detection method based on a large language model, which uses a pre-trained large language model to enhance time series reconstruction. The attention layer and the feed-forward neural network layer of the pre-trained language model contain most of the knowledge learned in language tasks. By calculating the anomaly score to obtain a threshold, the time series data can be automatically and accurately detected for anomalies. Description of the Drawings
[0053] Figure 1 is a schematic flowchart of a preferred embodiment of the present invention;
[0054] Figure 2 is a schematic diagram of generating a prompt statement in a preferred embodiment of the present invention;
[0055] Figure 3 is a schematic structural diagram of an anomaly detection model in a preferred embodiment of the present invention. Detailed Embodiments
[0056] The embodiments of the present invention will be described in detail below. The following embodiments are implemented on the premise of the technical solution of the present invention, and detailed implementation manners and specific operation processes are given. However, the protection scope of the present invention is not limited to the following embodiments.
[0057] The present invention designs an unsupervised time series anomaly detection method based on a large language model, as Figure 1 shown, the technical solution includes the following steps, specifically including:
[0058] S1: Preprocess the time series data, set a time window and divide the time series data into a training set and a test set;
[0059] S2: Extract the time series data features;
[0060] S3: Perform block and position embedding processing on the time window;
[0061] S4: Use the large language model as the backbone model and design an embedding layer and an attention layer to construct an anomaly detection model;
[0062] S5: Judge whether the time series data is abnormal based on the anomaly detection model.
[0063] Preferably, the preprocessing of the time series data in S1 includes:
[0064] Use a Gaussian filter to denoise the time series data, and the formula is as follows:
[0065]
[0066] Where y is the denoised time series data, σ is the standard deviation of the Gaussian filter, x[k] is the data point at position k in the time series, k is the position of the current time point, and i is the current time point;
[0067] The time series data is standardized using the following formula:
[0068]
[0069]
[0070] Where y i is the data point of the processed time series data, x i is the data point at the i-th time point in the time series, is the mean of x, s is the standard deviation, and n is the number of data points in the time series.
[0071] Preferably, setting the time window in S1 includes:
[0072] Convert the data points of the time series data into time windows. The time series data is represented as X = {x1, x2, …, x n}, and the local context window of length S is represented as W t = x t-S+1 , …, x t , and the window sequence data is F = {W1, W2, …, W t}, where the length of the time window W t is n, and X t ∈R m .
[0073] Preferably, S2 includes:
[0074] Use mean, variance, skewness, and kurtosis statistics as the characteristics of the time series data. The formula is as follows:
[0075]
[0076] Where μ is the mean, N is the number of samples of the time series data, σ 2 is the variance, sk is the skewness, and p is the kurtosis.
[0077] Specifically, such as Figure 2As shown, the waveform feature extraction for time series is achieved through fast Fourier transform, and data such as the frequency value corresponding to the maximum amplitude is obtained; among them, the prompt is a direct and effective method for activating specific tasks of large language models. Recent progress shows that other data patterns, such as images, can be seamlessly integrated as the prefix of the prompt, thus promoting effective reasoning based on these inputs. For enhancing the adaptability of large language models to downstream tasks in time series tasks. Use the statistical data features in the above steps as padding to dynamically generate prompt statements.
[0078] Preferably, S3 includes:
[0079] For the time window input, use each input channel x i , divide x i into multiple consecutive data blocks, integrate and process each data block, and generate a position encoding vector for the position t of each data block The i-th element of it is defined as follows:
[0080]
[0081]
[0082] In the formula, pos is the position, d model is the embedding dimension, and i is the dimension index of the embedding vector.
[0083] Preferably, S4 includes:
[0084] Construct an embedding layer based on the linear probing technique and use an Adapter method, including inserting a module in the Transformer model structure. In the attention layer, the Adapter method introduces an additional parameter matrix to adjust the output of the attention mechanism. The attention layer of the Transformer adopts a cross-attention mechanism, and the formula is as follows:
[0085] Q = XW Q , K = XW K , V = XW V
[0086]
[0087] MultiHead(Q, K, V) = Concat(h1, h2, …, h h )W O , where h i = Attention(Q i , K i , V i )
[0088] In the formula, WQ ,W K ,W V is a learnable parameter matrix, Q, K, and V are vectors derived from the input sequence through preprocessing, and d k is the number of columns of the Q, K matrices.
[0089] Specifically, in order to reprogram the input time series into a text prototype representation that is more naturally suitable for the capabilities of the language model, it is necessary to align the modalities of the time series and natural language to activate the time series understanding and reasoning capabilities of the pre-trained model. Since the word embedding layer of the original model contains a large number of parameters existing in language task training, and most of these parameters are not required in the time series task area, this will lead to problems such as reduced model training speed and reduced embedding accuracy. Therefore, it is necessary to reduce the useless parameters in this part when performing time series tasks. This solution uses an adapter method. Using an adapter can also learn multiple tasks with a relatively small number of parameters. Only a small number of parameters are required for one task, the training is faster, the memory occupied is less, and it is less likely to overfit for tasks with a small dataset.
[0090] Among them, in order to make the model adapt to time series data, it is marked with the PatchTST method, which adopts channel independence and patches. Channel independence regards multivariate time series as multiple univariate time series data, and then processes this data with a single model. The channel mixing model aims to directly utilize cross-channel data, while channel independence usually indirectly captures cross-channel interactions through weight sharing, thus providing superior performance. This is because channel mixing is often affected by limited data and overfitting.
[0091] In addition, after word embedding the time series, it is combined with the prompt embedding as the new input of the pre-trained model for model training. The length of the prompt input is marked, and after training is completed, the prompt part is discarded to obtain the corresponding time series for the next anomaly detection task.
[0092] Preferably, S5 includes:
[0093] S501: Train the anomaly detection model using the mean squared error of the actual values;
[0094] S502: Input the time series data X into the trained anomaly detection model to output the anomaly score;
[0095] S503: Determine the threshold based on the super-threshold peak method;
[0096] S504: Compare the anomaly score with the threshold to detect whether the timestamp is abnormal.
[0097] Preferably, S501 includes:
[0098] Minimize the difference between the reconstructed value and the actual value. Given the time series data X of length N as the input to the model, denoted as X = {x1, x2, …, x N}, the overall loss function of the anomaly detection model is expressed as follows:
[0099]
[0100] where Θ are the parameters of the anomaly detection model.
[0101] Preferably, S502 includes:
[0102] Input the time series data X into the trained anomaly detection model, and for each output time stamp x t , calculate the mean variance between it and the reconstructed value as the anomaly score, with the formula as follows:
[0103]
[0104] Preferably, S504 includes
[0105] For the anomaly score s i of each dimension and the corresponding dimension threshold POT(s t ), it is defined by the anomaly diagnosis label, with the formula as follows:
[0106] S i = 1(s i > POT(S t ))
[0107] where, if the time point i is greater than the threshold, then S i is 1, otherwise it is 0.
[0108] Specifically, for anomaly detection and diagnosis, only consider the data before the current time stamp, calculate the anomaly score of each dimension of the time stamp, and if this score is greater than a threshold, mark the time stamp as abnormal. The peak over threshold (POT) method is used to dynamically select the threshold. The POT method is a statistical method that uses "extreme value theory" to fit the data distribution with a generalized Pareto distribution and identify appropriate risk values to dynamically determine the threshold.
[0109] The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations based on the concept of the present invention without creative labor. Therefore, all technical solutions that can be obtained by those skilled in the art in the technical field based on the concept of the present invention through logical analysis, reasoning, or limited experiments on the basis of the prior art should fall within the protection scope determined by the claims.
Claims
1. An unsupervised time series anomaly detection method based on large language models, characterized in that, Including: S1: Preprocess the time series data, set a time window, and divide the time series data into a training set and a test set; S2: Extract the features of the time series data; S3: Perform chunking and positional embedding processing on the time window; S4: Use a large language model as the backbone model and design an embedding layer and an attention layer to construct an anomaly detection model; S5: Based on the anomaly detection model, determine whether the time series data is abnormal.
2. The unsupervised time series anomaly detection method based on a large language model according to claim 1, wherein The preprocessing of the time series data in S1 includes: Use a Gaussian filter to denoise the time series data, and the formula is as follows: In the formula, y is the denoised time series data, σ is the standard deviation of the Gaussian filter, x[k] is the data point at position k in the time series, k is the position of the current time point, and i is the current time point; Perform normalization processing on the time series data, and the formula is as follows: where y i is the data point of the processed time series data, x i is the data point of the i-th time point in the time series, is the average value of x, s is the standard deviation, and n is the number of data points in the time series.
3. The unsupervised time series anomaly detection method based on a large language model according to claim 1, wherein The setting of the time window in S1 includes: Convert the data points of the time series data into a time window, and the time series data is represented as X = {x1, x2, …, x n}, represent the local context window of length S as W t = x t-S+1 , …, x i , and the window sequence data is F = {W1, W2, …, W t}, where the length of the time window W t is n, and X t ∈ R m .
4. The unsupervised time series anomaly detection method based on a large language model according to claim 1, wherein S2 includes: Use mean, variance, skewness, and kurtosis statistics as the features of the time series data, and the formula is as follows: where μ is the mean, N is the number of samples of the time series data, σ 2 is the variance, sk is the skewness, and p is the kurtosis.
5. The unsupervised time series anomaly detection method based on a large language model according to claim 1, wherein S3 includes: For each input channel x of the time window input i , divide x i into multiple consecutive data blocks, integrate and process each data block, and generate a position encoding vector for the position t of each data block The i-th element of which is defined as follows: where pos is the position, d model is the embedding dimension, and i is the dimension index of the embedding vector.
6. The unsupervised time series anomaly detection method based on a large language model according to claim 1, characterized in that S4 includes: Construct an embedding layer based on linear probing technology and use an Adapter method, including inserting a module in the Transformer model structure. In the attention layer, the Adapter method introduces an additional parameter matrix to adjust the output of the attention mechanism. The attention layer of the Transformer uses a cross-attention mechanism, and the formula is as follows: Q = XW Q , K = XW K , V = XW V MultiHead(Q, K, V) = Concat(h1, h2, …, h h )W O , where h i = Attention(Q i , K i , V i ) where, W Q , W K , W V are learnable parameter matrices, Q, K, and V are vectors derived from the input sequence through preprocessing, and d k is the number of columns of the Q, K matrices.
7. A method for unsupervised time series anomaly detection based on a large language model according to claim 1, characterized in that, S5 includes: S501: Train the anomaly detection model using the mean squared error of the actual value; S502: Input the time series data X into the trained anomaly detection model and output an anomaly score; S503: Determine the threshold based on the super-threshold peak method; S504: Compare the anomaly score with the threshold to detect whether the timestamp is abnormal.
8. The unsupervised time series anomaly detection method based on a large language model according to claim 7, wherein S501 includes: Minimize the difference between the reconstructed value and the actual value, given the time series data X of length N as the input to the model, denoted as X = {x1, x2, …, x N}, and the overall loss function of the anomaly detection model is expressed as follows: In the formula, Θ is the parameter of the anomaly detection model.
9. The unsupervised time series anomaly detection method based on a large language model according to claim 7, wherein S502 includes: Input the time series data X into the trained anomaly detection model, and for each timestamp x outputted t , calculate its mean and variance with the reconstructed value as the anomaly score, and the formula is as follows:
10. The unsupervised time series anomaly detection method based on a large language model according to claim 1, wherein S504 includes For the anomaly score s for each dimension i and the threshold POT(s t ) for the corresponding dimension, which is defined by the anomaly diagnosis label, the formula is as follows: S i = 1(s i > POT(s t )) where, if the time point i is greater than the threshold, S i is 1, otherwise it is 0.