Lightweight time sequence anomaly detection method combining point-level and sequence-level reconstruction association

Abstract

The invention discloses a point-level and sequence-level reconstruction association-combined lightweight time sequence anomaly detection method, which comprises the following steps of: firstly, constructing a lightweight pure MLP double-branch reconstruction network by using two MLPs with a small number of parameters as a skeleton to quickly learn local relevance and global relevance of each timestamp; the difference between the local association and the global association is used as reconstruction association; then, simultaneously executing relevance reconstruction learning from two complementary perspectives of a point level and a sequence level, and making up for the defects caused by a lightweight architecture through complementary learning; and finally, providing an anomaly scoring mechanism based on reconstruction association, combining point-level reconstruction association with sequence-level reconstruction association, and accurately identifying explicit anomalies and implicit anomalies in the time sequence. According to the lightweight time sequence anomaly detection method based on the pure MLP architecture, only four two-layer MLPs are utilized to achieve high speed and high precision, and the lightweight time sequence anomaly detection method can be easily deployed on Internet of Things edge equipment with limited resources.

Description

Technical Field

[0001] The present invention relates to the technical field of time series anomaly detection, and in particular to a lightweight time series anomaly detection method combining point-level and sequence-level reconstruction associations. Background Art

[0002] Time series present some new characteristics: features are more complex and changeable, variables are more and more coupled, and sequence labels are scarcer. These characteristics lead to poor performance or even inability of traditional time series anomaly detection models. Specifically, traditional time series anomaly detection models based on clustering, statistics, and machine learning have difficulty in accurately extracting complex features in time series. At the same time, traditional time series anomaly detection models based on supervised learning and semi-supervised learning have difficulty dealing with scarce labels. Therefore, time series anomaly detection models based on unsupervised deep learning, such as models based on convolutional neural networks (CNN), Transformer, and recurrent neural networks (RNN), have been a research hotspot in recent years. However, existing time series anomaly detection models based on unsupervised deep learning have ushered in two new challenges.

[0003] (1) Challenge 1: Existing anomaly scoring schemes are not powerful enough. Typically, a time series anomaly detection model requires an anomaly scoring scheme to evaluate the observations at each timestamp and determine whether it is an anomaly. Existing scoring schemes can be divided into reconstruction error-based, prediction error-based, and mixed error-based. Specifically, the reconstruction error-based scoring scheme usually uses the reconstruction error of each timestamp as the anomaly score. This scheme is fast and simple, but ignores the contextual information of each timestamp and its neighbors. Therefore, this scheme can work with shallow neural networks to achieve high timeliness, but cannot guarantee accuracy. The prediction error-based scoring scheme uses the model's prediction error as the anomaly score. This scheme takes into account the contextual information of each timestamp and is very compatible with deep time series anomaly detection models. Although this type of scheme achieves high accuracy, its timeliness is poor. The mixed error-based scheme combines the above two methods, so the model is more complex and the timeliness is not satisfactory. Therefore, building a more powerful anomaly scoring mechanism that can balance high speed and high accuracy is a key challenge for existing time series anomaly detection models.

[0004] To address the first challenge, a new anomaly scoring scheme based on correlation difference was proposed in 2022. The scheme believes that "compared with normal points, anomalies have stronger correlation with local neighbors and weaker correlation with global neighbors". In the past two years, many time series anomaly detection models based on correlation difference have been proposed and proven to have excellent performance. Usually, the detection model based on correlation difference constructs two branches to learn the correlation of each timestamp with local neighbors and the correlation with global neighbors respectively, and then calculates the difference between the two correlations to determine whether a timestamp is anomaly. For example, the AnomalyTransformer model first uses a Gaussian kernel to learn the correlation between a timestamp and its local neighbors; at the same time, a multi-head attention mechanism is used to learn the correlation between the timestamp and its global neighbors; finally, a minimum-maximum training strategy is designed to amplify the difference between the two correlations. On this basis, the MAN-QSM model adds a mask learning mechanism to generate correlation differences more accurately. Similarly, the DCdetector model designs a two-branch attention contrastive learning architecture, using two multi-head attentions to learn the local correlation and global correlation of each timestamp. The Dual-TF model considers fine-grained association differences in both the time domain and the frequency domain to more accurately identify anomalies. The SiET model further learns the spatial association of multiple variables and combines the spatial and temporal association differences for anomaly detection. On this basis, the STAT model further considers the order of temporal feature learning and spatial feature learning to better capture spatial and temporal features and achieve more accurate anomaly detection. The GLAD model attempts to learn spatial and temporal association differences from both the point level and the sequence level.

[0005] In general, the anomaly scoring scheme based on correlation difference can better learn the long-term and short-term contextual information of each timestamp, thereby effectively identifying anomalies in time series. However, correlation calculation is usually a fine-grained operation, which causes these models to take a long time to train and detect, and also requires a lot of CPU / GPU resources and storage resources. Therefore, in order to build a time series anomaly detection model based on correlation difference with high time efficiency, it is necessary to abandon the traditional huge architecture and fine-grained computing mode, and instead adopt a lightweight, shallow architecture with fewer parameters and a coarse-grained correlation computing scheme.

[0006] (2) Challenge 2: It is difficult to achieve both high timeliness and high precision. With the rapid development of industrial intelligence, the supervision granularity of underlying equipment has become more fine-grained, which forces time series anomaly detection models to have both high timeliness and high precision. Specifically, an ideal time series anomaly detection model should detect anomalies as quickly and accurately as possible and provide warnings, thereby reserving time for subsequent intelligent operation and maintenance. However, high speed and high precision are often not easy to achieve at the same time. For example, time series data often exhibits complex characteristics such as periodicity, seasonality, and scale. In order to accurately capture features, existing deep time series anomaly detection models are keen to use deep structures and large parameters to enhance their feature extraction capabilities. In this case, these models require a lot of computing and storage resources, as well as a lot of training and detection time. Therefore, these models cannot achieve high speed and high timeliness. On the contrary, although traditional machine learning-based and statistical learning-based models have fast speed, their accuracy is not satisfactory. Therefore, balancing high speed and high progress is the difficulty of existing time series anomaly detection.

[0007] To address the second challenge, in recent years, many deep models for time series anomaly detection have adopted lightweight MLP networks to replace traditional time-consuming or resource-consuming modules, thereby achieving the purpose of compressing the model, reducing model parameters, and improving model timeliness. For example, the PatchAD model uses MLP to replace the attention module in the Transformer network, thereby reducing the time spent on anomaly detection. Similarly, the ANNet model uses MLP to optimize LSTM units to improve model speed. In the field of time series prediction, the use of MLP networks is more in-depth. Some excellent time series analysis models based on pure MLP have been proposed and have excellent performance. For example, the RTSF model has conducted a large number of experiments to prove that the MLP network can replace the CNN network and the attention network. Based on this finding, the Solar-Mixer model and the TS-Mix model try to stack multiple MLP layers to build an All-MLP architecture to quickly and accurately predict time series. In order to improve the prediction performance, the HDMiXer model designs a multi-layer MLP network to learn the temporal and spatial features between multiple variables. The STHMLP model further develops a hierarchical MLP network to learn multi-scale spatiotemporal features. In order to further utilize the frequency domain information, the FreTS model employs and only employs a shallow MLP network to directly learn the characteristics of the time series in the frequency domain. Inspired by the FreTS model, the FITS model directly uses the MLP network to simulate the interpolation process in the frequency domain to achieve accurate time series prediction. It is worth noting that the FITS model has less than 10k parameters, achieving the dual goals of high timeliness and high accuracy.

[0008] In summary, the simple MLP network is a good choice for improving the timeliness of the model. However, the research on MLP networks in time series anomaly detection is still in its early stages and is not as in-depth as in the field of time series prediction. Specifically, the existing MLP-based time series anomaly detection model only uses MLP to compress the model to improve timeliness while ensuring high accuracy. However, the compressed model also has a complex structure and large parameters, and the improvement in timeliness is not satisfactory enough. Summary of the invention

[0009] In response to the above-mentioned technical problems to be solved, the present invention provides a lightweight time series anomaly detection method that combines point-level and sequence-level reconstruction associations, and utilizes a lightweight pure MLP architecture with only a shallow structure and a small number of parameters and a customized reconstruction difference scoring strategy to simultaneously achieve high-precision and high-timeliness anomaly detection.

[0010] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0011] A lightweight time series anomaly detection method that combines point-level and sequence-level reconstruction associations. The anomaly detection method first uses two two-layer MLPs as a skeleton to construct a lightweight pure MLP dual-branch reconstruction network to learn the local and global associations of each timestamp, and uses the difference between the local and global associations as the reconstruction association; then, correlation reconstruction learning is performed simultaneously from two complementary perspectives, point-level and sequence-level, to make up for the inconsistency caused by the lightweight architecture through complementary learning. Finally, an anomaly scoring mechanism based on reconstruction association is designed, which combines point-level reconstruction association and sequence-level reconstruction association. combine, Accurately identify explicit and implicit anomalies in time series.

[0012] The anomaly detection method uses the idea that "compared with normal points, anomalies often have strong correlations with local neighbors and only weak correlations with global neighbors" to guide the realization of lightweight anomaly detection based on a shallow pure MLP architecture.

[0013] As a further improvement of the above technical solution:

[0014] Preferably, the anomaly detection method specifically includes the following contents:

[0015] Step S1, point-level reconstruction association learning: construct a lightweight first two-branch reconstruction network consisting of two two-layer MLPs, learn the local association and global association of each timestamp from a point-level perspective, and generate point-level reconstruction associations;

[0016] Step S2, sequence-level reconstruction association learning: construct another lightweight pure MLP two-branch network to complement the first two-branch reconstruction network, learn the local and global associations of each timestamp from a sequence-level perspective, and generate sequence-level reconstruction associations;

[0017] Step S3, anomaly scoring based on reconstructed associations: Combine the point-level and sequence-level reconstructed associations to generate an anomaly score for each timestamp and decide whether the current timestamp is an anomaly.

[0018] Preferably, the step S1 specifically includes the following steps:

[0019] S1-1, point-level sampling: First, multiple sequences of length WIN_SIZE are randomly extracted in parallel from the time series; second, each sequence is further divided into N1 non-overlapping segments, each segment contains N2 timestamps, and the product of N1 and N2 is equal to WIN_SIZE; then, the N1 segments are stacked to form a 2D matrix of size N1×N2; finally, the local neighbors and global neighbors of each timestamp are quickly generated from the 2D matrix; for a current timestamp in the i-th row and j-th column, the local neighbors are composed of other timestamps in the i-th row, and the global neighbors are other timestamps in the j-th column;

[0020] S1-2, local association learning: a two-layer MLP network with is used as the first branch of a lightweight two-branch reconstruction network to learn the association between each timestamp and its local neighbors;

[0021] S1-3, global association learning: another two-layer MLP is used as the second branch of the lightweight two-branch reconstruction network to learn the association of each timestamp with its global neighbors;

[0022] S1-4, loss function: The local association learning process and the global association learning process constitute two contrast branches, and the MSE function is used as the loss function of contrast learning to align the reconstruction values ​​of the two branches.

[0023] Preferably, in S1-2 and S1-3, the two-layer MLP network includes an input layer with N2-1 neurons and a hidden layer with d neurons, the default value of the hidden layer is 64 or 128, and an output layer with 1 neuron; the input layer and the hidden layer constitute a first fully connected network with (N2-1)×d trainable parameters; the hidden layer and the output layer form a second fully connected network with d×1 trainable parameters; the hidden layer uses dropout and ReLU activation functions.

[0024] Preferably, the step S2 specifically includes the following steps:

[0025] S2-1, sequence-level sampling: First, multiple sequences of length W_SEQ are randomly and parallelly extracted from the time series; then, each sequence is divided into N3 non-overlapping segments; at the same time, each segment is divided into N4 non-overlapping sub-segments, each sub-segment contains N5 timestamps, and the product of N3, N4 and N5 is equal to W_SEQ; then, the N3 segments are stacked to form a 2D matrix of size N3×(N4×N5); finally, the sequence-level local neighbors and global neighbors of each timestamp are generated from the 2D matrix. For a current timestamp located in the i-th row and the j-th column, the sub-segment of the sequence to which it belongs is used as the replacement sequence, the other sub-segments in the row i are used as local neighbors, and the sub-segments corresponding to other timestamps in the column j are used as global neighbors;

[0026] S2-2, local association learning: a two-layer MLP network is used as the first reconstruction branch to learn the association between each timestamp and its local neighbors from a sequence perspective;

[0027] S2-3, global association learning: another two-layer MLP network is used as the second reconstruction branch to learn the association between each timestamp and its global neighbors from a sequence perspective;

[0028] S2-4, loss function: The local association learning process and the global association learning process constitute two contrast branches, and the MSE function is used as the contrast learning loss function.

[0029] Preferably, in S2-2 and S2-3, the two-layer MLP network includes an input layer with N4-1 neurons and a hidden layer with d neurons, the default value of the hidden layer is 64 or 128, and an output layer with N5 neurons; the input layer and the hidden layer constitute a first fully connected network; the hidden layer and the output layer form a second fully connected network; the hidden layer uses dropout and ReLU activation functions, and the two-layer MLP network contains N4×N5×d training parameters.

[0030] Preferably, in step S3, specifically:

[0031] S3-1, set a timestamp x in the test set t Input into point-level and sequence-level reconstruction association learning at the same time to perform parallel learning;

[0032] S3-2, performing parallel correlation calculation on M variables of the t-th timestamp in point-level reconstruction correlation learning and sequence-level reconstruction correlation learning; after the calculation is completed, generating M reconstruction correlation values ​​of the t-th timestamp respectively;

[0033] S3-3, calculate the average value of the M reconstruction association values ​​in each reconstruction association learning, and generate the point-level reconstruction association of timestamp t and sequence-level reconstruction

[0034] S3-4, the anomaly score of the t-th timestamp is generated by combining the point-level reconstruction association and the sequence-level reconstruction association, which is:

[0035] Score(x t )=α×Score P (x t )+(1-α)×Score S (x t ) (13)

[0036] Among them, α is a preset parameter, ranging from [0 to 1]; when α = 0, only the sequence-level reconstruction association is considered; when α = 1, only the point-level reconstruction association is considered; represents the point-level reconstruction association on the mth variable, represents the sequence-level reconstruction association on the mth variable;

[0037] S3-5, after obtaining the anomaly score, determine whether the tth timestamp is an anomaly using formula (2):

[0038]

[0039] Where λ is a preset threshold ranging from 0 to 1; if Indicates that the tth timestamp is abnormal; otherwise, the tth timestamp is normal.

[0040] The lightweight time series anomaly detection method for combining point-level and sequence-level reconstruction associations provided by the present invention has the following advantages over the prior art:

[0041] (1) The lightweight time series anomaly detection method of the present invention, which combines point-level and sequence-level reconstruction associations, abandons the traditional solution of building a cumbersome architecture with deep structures and large parameters, and instead designs a lightweight pure MLP architecture consisting of only two layers of structure and a small number of parameters to achieve anomaly detection with both high speed and high accuracy. Specifically, the method employs two two-layer MLPs with only a small number of parameters from both the point-level and sequence-level perspectives to construct a lightweight two-branch reconstruction network for learning the point-level and sequence-level reconstruction associations of each timestamp. This is the first lightweight time series anomaly detection model based on pure MLP.

[0042] (2) The lightweight time series anomaly detection method based on joint point-level and sequence-level reconstruction associations proposed in this paper proposes an anomaly scoring scheme based on reconstruction associations, which combines point-level and sequence-level reconstruction associations to accurately identify explicit and implicit anomalies. This is also the first work that attempts to integrate the reconstruction error-based scoring scheme and the association difference-based scoring scheme. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of explicit and implicit anomalies in time series.

[0044] Figure 2 It is a framework diagram of anomaly detection in the present invention.

[0045] Figure 3 It is a schematic diagram of point-level sampling and sequence-level sampling in the present invention.

[0046] Figure 4 It is a schematic diagram comparing the abnormal scoring scheme based on reconstructed association in the present invention with the traditional scoring scheme.

[0047] Figure 5 It is a process diagram of abnormal scoring based on reconstructed association in the present invention.

[0048] FIG6( a ) is a visualization diagram of anomaly detection within timestamps 200-1200 of the first variable of the MSL dataset in the experimental verification of the present invention.

[0049] FIG6( b ) is a visualization of anomaly detection within the 26400-27400 timestamp of the univariate MITDB dataset in the experimental verification of the present invention.

[0050] Figure 7 It is the deployment result of ten comparison models on the Raspberry Pi 4b edge device in the experimental verification of this invention. DETAILED DESCRIPTION

[0051] The specific embodiments of the present invention are described in detail below. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0052] The lightweight time series anomaly detection method of the present invention combines point-level and sequence-level reconstruction associations, and uses two two-layer MLPs to construct a lightweight dual-branch reconstruction network from the point-level and sequence-level perspectives, thereby generating point-level and sequence-level reconstruction associations for each timestamp. Then, the two reconstruction associations are combined to perform fast and accurate time series anomaly detection.

[0053] The lightweight time series anomaly detection method of the present invention combines point-level and sequence-level reconstruction associations. The principle is: M sensors are used to observe a dynamic system T times, and the collected data is a time series, which can be expressed as A time series consists of two dimensions: time dimension and variable dimension. For the time dimension, where x t Represents the observation value at the M sensors at the tth time stamp. For the variable dimension, where xm represents the observation value of the mth sensor at all T timestamps. In the present invention, a time series X is divided into an unlabeled training set X 1 and a labeled test set X 2 ; Then, using the unlabeled training set X 1 To train the constructed lightweight time series anomaly detection model; then use the trained model to test the test set X 2 Anomaly scoring is performed for each timestamp in:

[0054] Score t =TSAD(x t ) (1)

[0055] in Yes X 2 The observed values ​​of the M variables at the tth time stamp;

[0056] Finally, the score of the anomaly score is used to decide whether the tth timestamp is an anomaly, which is defined as follows:

[0057]

[0058] Where λ is a preset threshold ranging from 0 to 1; if Indicates that the tth timestamp is abnormal; otherwise, the tth timestamp is normal.

[0059] The lightweight time series anomaly detection method of the present invention combines point-level and sequence-level reconstruction associations, and performs detection by constructing an unsupervised lightweight time series anomaly detection model (referred to as LFTSAD) for joint point-level and sequence-level reconstruction associations. LFTSAD abandons the traditional deep architecture with huge parameters, and only uses two two-layer MLPs to construct a lightweight pure MLP dual-branch reconstruction network, and learns the reconstruction association of each timestamp from two perspectives of point level and sequence level. In addition, LFTSAD designs a novel scoring mechanism based on reconstruction association, which improves the model accuracy by combining point-level and sequence-level reconstruction associations, and solves the shortcomings of the lightweight architecture.

[0060] The lightweight time series anomaly detection method of the present invention combines point-level and sequence-level reconstruction associations. The framework of the LFTSAD model is as follows: Figure 2 As shown, including the following:

[0061] Step S1: Point-level reconstruction association learning

[0062] To identify explicit anomalies in time series ( Figure 1 (a)), a lightweight pure MLP two-branch network is constructed to learn the local and global associations of each timestamp from a point-level perspective, and generate the final point-level reconstruction association, as shown in Figure 2 (a) shown.

[0063] Firstly, a sampling strategy is designed to generate local neighbors and global neighbors for each timestamp. Then, a lightweight two-branch reconstruction network is constructed using two two-layer MLPs to simultaneously reconstruct the same timestamp using local neighbors and global neighbors respectively. Finally, the difference between the reconstructed values ​​of the two branches is the point-level reconstruction association of each timestamp.

[0064] The specific steps include:

[0065] S1-1, point-level sampling

[0066] Design a sampling strategy to generate local neighbors and global neighbors for each timestamp. In the sampling strategy, adjacent timestamps are considered local neighbors, while timestamps that are far apart are considered global neighbors. Figure 3 As shown in (a), the detailed sampling process is as follows:

[0067] First, multiple sequences of length WIN_SIZE are randomly extracted in parallel from the time series. Second, each sequence is divided into N1 non-overlapping segments, each segment contains N2 timestamps, and the product of N1 and N2 is equal to WIN_SIZE. Then, the N1 segments are stacked together to form a 2D matrix of size N1×N2. Finally, the local neighbors and global neighbors of each timestamp are quickly generated from this 2D matrix. For example, for a timestamp in the i-th row and j-th column, its local neighbors consist of other timestamps in the i-th row, and its global neighbors are other timestamps in the j-th column.

[0068] This sampling strategy has the advantages of high speed and low resource consumption. Furthermore, in a 2D matrix, all timestamps in the same row share the same set of local neighbors, and all timestamps in the same column share the same set of global neighbors. Therefore, all timestamps in the 2D matrix can share the same storage space, which greatly reduces resource consumption. Moreover, since there is no need to calculate the local and global neighbors for each timestamp separately, the sampling speed is greatly accelerated. In addition, when processing multivariate time series, the sampling process of each variable is completely independent and can be executed in parallel to further speed up.

[0069] S1-2, Local Association Learning

[0070] Association calculation is usually a fine-grained operation, which requires evaluating the association between each timestamp and each neighbor. For example, if a timestamp has N2 neighbors, N2 associations need to be calculated. However, the MLP lightweight network of this embodiment cannot complete this fine-grained calculation. To solve this problem, all neighbors are used to reconstruct each timestamp. If a timestamp can be accurately reconstructed, it means that it has a strong association with its neighbors; otherwise, the association is weak. This scheme reduces the fine-grained association calculation to one operation for each timestamp, greatly simplifying the calculation process.

[0071] A two-layer MLP network is used as the first branch of the MLP two-branch network to learn the association of each timestamp with local neighbors. This two-layer MLP network includes an input layer with N2-1 neurons, a hidden layer with d neurons, and an output layer with 1 neuron. The input layer and the hidden layer form the first fully connected network with (N2-1)×d trainable parameters. The hidden layer and the output layer form the second fully connected network with d×1 trainable parameters. To speed up training, the hidden layer uses dropout and ReLU activation functions.

[0072] Take the mth variable in the time series X For example, the tth timestamp is The local neighbors are where the timestamps from t-N2 to t share the same local neighbors.

[0073] The local correlation calculation process of the t-th timestamp is as follows:

[0074] S1-2-1, input N2-1 local neighbors into the first fully connected network to learn relevant features.

[0075]

[0076] in is the output of the first fully connected network, are the parameters of the first fully connected network; ReLU is the activation function; LN represents the local neighbors of timestamp t.

[0077] S1-2-2, input h1 into the second fully connected network to reconstruct As shown below:

[0078]

[0079] in is the local reconstruction value, are the parameters of the second fully connected network.

[0080] S1-3, global association learning

[0081] Similar to local association learning, another two-layer MLP is used as the second branch of the lightweight two-branch network to learn the association of each timestamp with its global neighbors. This MLP has the same structure as the first MLP, only including two fully connected networks and N1×d parameters.

[0082] Furthermore, with the mth variable For example, the tth timestamp is The global neighbors are where all timestamps share the same global neighbors.

[0083] The global correlation calculation process of the t-th timestamp is as follows:

[0084] S1-3-1, input N1-1 global neighbors into the first fully connected network to learn relevant features, as shown below:

[0085]

[0086] in is the output, are the parameters of the first fully connected network.

[0087] S1-3-2, input the learned h2 into the second fully connected network to reconstruct

[0088]

[0089] in is the global reconstruction value, is the second fully connected network parameter.

[0090] All timestamps of the same variable share two MLPs of the lightweight two-branch network. Moreover, all variables of the time series also share these two MLP networks.

[0091] S1-4, loss function

[0092] The local association learning process and the global association learning process constitute two contrast branches, and the MSE function is used as the contrast learning loss to align the reconstruction values ​​of the two branches:

[0093]

[0094] Among them, L P represents point-level contrast loss, M is the number of variables, WIN_SIZE is the window size of a sample, is the local reconstructed value of the mth variable at the tth timestamp, is the global reconstruction value.

[0095] Step S2: sequence-level reconstruction association learning.

[0096] To identify explicit anomalies in time series ( Figure 1 (b)), this embodiment constructs another lightweight pure MLP dual-branch network as a supplement to the first dual-branch network, learns the local association and global association of each timestamp from the sequence level perspective, and generates the final sequence-level reconstruction association, as shown in Figure 2 (b) as shown.

[0097] Furthermore, first, a sequence-level sampling strategy is adopted to generate sequence-level local and global neighbors for each timestamp; then, a lightweight two-branch network is constructed to simultaneously reconstruct alternative subsequences of each timestamp using local and global neighbors; finally, the difference between the reconstructed values ​​of the two subsequences is the sequence-level reconstructed association of each timestamp.

[0098] In an implicit anomaly that lasts for multiple timestamps, some timestamps have similar neighbor association patterns as normal timestamps, while other timestamps have significantly different neighbor association patterns from normal timestamps. This means that only using point-level reconstruction association learning cannot accurately and perfectly identify all abnormal timestamps. To solve this problem, this embodiment constructs a sequence-level reconstruction association learning process as a supplement to the point-level reconstruction association learning process to accurately identify implicit anomalies.

[0099] S2-1, sequence level sampling

[0100] The sequence-level sampling strategy designed in this embodiment first randomly and in parallel extracts multiple sequences of length W_SEQ from the time series X; then, each sequence is divided into N3 non-overlapping segments; at the same time, each segment is further divided into N4 non-overlapping sub-segments, each of which contains N5 timestamps, and the product of N3, N4 and N5 is equal to W_SEQ; then, the N3 segments are stacked to form a 2D matrix of size N3×(N4×N5), as shown in Figure 3 As shown in (b); finally, the sequence-level local and global neighbors of each timestamp can be quickly generated from this 2D matrix.

[0101] For example, for a timestamp located at the i-th row and the j-th column, the subsegment to which it belongs is used as the replacement sequence, other subsegments in the row i are used as its local neighbors, and subsegments corresponding to other timestamps in the column j are used as its global neighbors.

[0102] S2-2, Local Association Learning

[0103] Similar to the point-level learning process, a two-layer MLP network is used as the first branch to learn the association of each timestamp with local neighbors from a sequence perspective. Unlike the previous MLP, this MLP has (N4-1)×N5 neurons in the input layer, d neurons in the hidden layer, and N5 neurons in the output layer. The input layer and the hidden layer form the first fully connected network, and the hidden layer and the output layer form the second fully connected network. The MLP network contains two fully connected networks and N4×N5×d parameters.

[0104] Furthermore, with the mth variable For example, the tth timestamp is Its replacement subsequence is The local neighbors are All timestamps in the replacement subsequence share the same local neighbors. The sequence-level local association learning process for the t-th timestamp is as follows:

[0105] S2-2-1, input (N4-1)×N5 local neighbors into the first fully connected network to learn the associated features, as follows:

[0106]

[0107] in is the output of the first fully connected network, are the parameters of the first fully connected network.

[0108] S2-2-2, input h3 into the second fully connected network to reconstruct the alternative subsequence of the t-th timestamp

[0109]

[0110] in, is the local reconstruction value, are the parameters of the second fully connected network.

[0111] S2-3, global association learning

[0112] A two-layer MLP network is used as the second branch to learn sequence-level global correlation. In this MLP, the input layer has (N3-1)×N5 neurons, the hidden layer has d neurons, and the output layer has N5 neurons. The MLP has a total of N3×N5×d parameters.

[0113] Taking the tth timestamp of the mth variable as an example, its replacement subsequence is Its global neighbors are The sequence-level global association learning process for the t-th timestamp is as follows:

[0114] S2-3-1, input the (N3-1)×N5 global neighbors into the first fully connected network.

[0115]

[0116] in is the output of the first fully connected network, are the parameters of the first fully connected network.

[0117] S2-3-2, input h4 into the second fully connected network to reconstruct the alternative subsequence of the t-th timestamp.

[0118]

[0119] in is the global reconstruction value, are the parameters of the second fully connected network.

[0120] S2-4, the MSE function is used as the contrastive learning loss function.

[0121]

[0122] Where L S represents the sequence-level contrast loss, M is the number of variables, W_SEQ is the window size of a sample, is the local reconstruction value, is the global reconstruction value.

[0123] Step S3, scoring anomalies based on the reconstructed associations.

[0124] The anomaly scoring scheme of this embodiment combines point-level and sequence-level reconstruction associations to generate the final anomaly score for each timestamp and decide whether it is an anomaly. The scoring scheme of this embodiment attempts to integrate the scoring scheme based on reconstruction error and the scoring scheme based on association difference for the first time, so as to adapt it to the lightweight network architecture based on MLP, thereby serving the lightweight model of this embodiment. Figure 4 shown.

[0125] like Figure 5 As shown, the detailed process of the abnormal scoring scheme of this embodiment is as follows:

[0126] S3-1, set a timestamp x in the test set t Input into point-level and sequence-level reconstruction association learning at the same time to perform parallel learning;

[0127] S3-2, performing parallel correlation calculation on the M variables of the t-th timestamp in the point-level reconstruction correlation learning and the sequence-level reconstruction correlation learning. After the calculation is completed, M reconstruction correlation values ​​of the t-th timestamp are generated respectively.

[0128] S3-3, calculate the average value of the M reconstruction association values ​​in each reconstruction association learning, and generate the point-level reconstruction association of timestamp t and sequence-level reconstruction

[0129] S3-4, the anomaly score of the t-th timestamp is generated by combining the point-level reconstruction association and the sequence-level reconstruction association, which is:

[0130] Score(x t )=α×Score P (x t )+(1-α)×Score S (x t ) (13)

[0131] Among them, α is a preset parameter, ranging from [0 to 1]; when α = 0, only the sequence-level reconstruction association is considered; when α = 1, only the point-level reconstruction association is considered; represents the point-level reconstruction association on the mth variable, represents the sequence-level reconstruction association on the mth variable;

[0132] S3-5, after obtaining the anomaly score, determine whether the tth timestamp is an anomaly using formula (2).

[0133] The unsupervised lightweight time series anomaly detection method that combines point-level and sequence-level reconstruction associations proposed in the present invention abandons the traditional solution of building a bulky architecture with a deep structure and large parameters, and instead designs a lightweight architecture consisting of a shallow structure and a small number of parameters to balance high speed and high precision. First, a lightweight dual-branch reconstruction network is constructed using two MLPs with only two layers and a small number of parameters to quickly learn the local and global correlations of each timestamp, and the difference between the local and global correlations is used as the reconstruction association. Secondly, based on the dual-branch reconstruction network of the lightweight pure MLP, two complementary point-level and sequence-level reconstruction learnings are designed to improve model accuracy and reduce the deficiencies brought by the lightweight architecture. Finally, an anomaly scoring mechanism based on reconstruction associations is proposed, which combines point-level reconstruction associations with sequence-level reconstruction associations to more accurately identify anomalies.

[0134] Experimental verification

[0135] (1) Experimental data

[0136] We selected 13 publicly available real-world time series datasets for experiments, including 6 univariate datasets and 7 multivariate datasets, as shown in Table 1. These datasets span multiple fields, such as industry, finance, medical care, manufacturing, etc., to ensure the fairness and impartiality of the experimental results.

[0137] 1) Multivariate datasets. The MSL dataset captures the operating status of multiple sensors or controllers on the Mars rover. The GECCO dataset stores drinking water quality data from multiple IoT sensors. The SWAN dataset contains temporal data from the solar photospheric vector magnetogram in the Spaceweather HMI active region patch series. The PSM dataset contains 25-dimensional monitoring information from eBay Server Machines. The SMAP dataset presents soil samples and telemetry data collected by the Mars rover. The SMD dataset stores resource utilization tracking records of 28 servers over a five-week period. The SWaT dataset consists of 51-dimensional data collected by multiple sensors in public water treatment infrastructure. In summary, these seven datasets cover the fields of aviation, commerce, water treatment, and telemetry.

[0138] 2) Univariate datasets. The ECG dataset includes electrocardiogram data with abnormal ventricular premature beats. The MITDB dataset includes 48.5 hours of dual-channel dynamic ECG recording excerpts. The SVDB dataset stores 78 half-hour ECG records with supraventricular arrhythmia abnormalities. The UCR dataset is a life sensor time series with a single anomaly. The UCR-AUG dataset is an enhanced data of the original UCR dataset. The Occupancy dataset is a binary classification dataset based on temperature, humidity, light, and CO2 levels. In summary, these six datasets cover fields such as nature, environment, and medicine.

[0139] Table 1 Experimental dataset

[0140]

[0141]

[0142] (2) Comparison of models and evaluation indicators

[0143] 1) Comparison model: In order to verify the effectiveness of the LFTSAD model of this embodiment, nine representative anomaly detection models were selected as competitors from three aspects: model structure, scoring scheme, and use of MLP network, as listed in Table 2. From the perspective of network structure, DCdetector, PatchAD, AnomalyTrans, DTAAD, TranAD, USAD, MSCRED, MTGFLOW, etc. are all deep time series anomaly detection models with multi-layer structures and a large number of parameters. In contrast, IForest and LFTSAD are time series anomaly detection models with fewer parameters and shallow structures. From the perspective of anomaly scoring schemes, LFTSAD, AnomalyTrans, DCdetector, and PatchAD utilize anomaly scoring schemes based on association differences. Other models rely on scoring schemes based on reconstruction errors or prediction errors. From the perspective of whether the MLP network is used, both LFTSAD and PatchAD use the MLP network as the backbone, of which LFTSAD is a lightweight network and PatchAD is a deep network.

[0144] 2) Evaluation indicators: In order to comprehensively evaluate the experiment, four widely used evaluation indicators are selected: accuracy (ACC), precision (Precision), recall (Recall) and F-Score. Specifically, ACC represents the ratio of correctly predicted samples to the total number of samples. Precision indicates how many of the timestamps predicted as normal are actually normal. Recall describes how many timestamps that are correctly predicted are actually normal. F-Score combines Precision and Recall to provide a more comprehensive evaluation. The value range of these four evaluation indicators is [0~1]. The higher the value, the better the performance of time series anomaly detection.

[0145] Table 2 Comparison models

[0146]

[0147]

[0148] (3) Accuracy experiment

[0149] Ten models were experimented on six univariate datasets and seven multivariate datasets, and each experimental result is the average of five identical experiments.

[0150] 1) Performance on univariate datasets: Table 3 lists the anomaly detection accuracy of 10 models on 6 univariate datasets. It is not difficult to find the following observations from the table.

[0151] I. Focus on comprehensive performance: LFTSAD performs best, followed by DTAAD, PatchAD, AnomalyTrans and DCdetector, and the worst are TranAD, IForest, USAD, MSCRED, and MTGFLOW. For example, on the MITDB dataset, LFTSAD has an ACC of 0.9963 and an F-Score of 0.9314, while TranAD has only an ACC of 0.8370 and an F-Score of 0.8850.

[0152] II. Focus on stability: LFTSAD performs the most stably on the four evaluation metrics, followed by DTAAD, PatchAD, and AnomalyTrans. For example, on the ECG and UCR datasets, LFTSAD presents 3 first and 1 second stable rankings.

[0153] III. Considering anomaly scoring schemes: Time series anomaly detection models based on correlation differences, such as LFTSAD, DCdetector, AnomalyTrans, and PatchAD, perform better than models based on other scoring schemes. For example, on the ECG dataset, DCdetector and AnomalyTrans outperform DTAAD (accuracy 0.4814) and USAD (accuracy 0.4830).

[0154] IV. Comparison of model structures: Deep models (PatchAD, AnomalyTrans, DCdetector, TranAD, USAD, MSCRED, MTGFLOW) generally outperform shallow models (IForest). However, the shallow LFTSAD surpasses multiple deep models. For example, on the SVDB dataset, the accuracy of PatchAD is 0.9245, the accuracy of LFTSAD is 0.6905, the accuracy of DTAAD is 0.6809, and the accuracy of IForest is only 0.4409.

[0155] Table 3 Performance of multiple models on univariate datasets

[0156]

[0157]

[0158]

[0159] 2) Performance on multivariate datasets: Table 4 shows the anomaly detection performance of the 10 models on 7 multivariate datasets. Several similar results can be drawn from the table.

[0160] I. Focus on comprehensive performance: LFTSAD has always performed at a high level, followed by DCdetector, AnomalyTrans and MSCRED. For example, on the MSL dataset, LFTSAD topped the list with an ACC of 0.9907 and an F-Score of 0.9658, while IForest only had an ACC of 0.3764 and an F-Score of 0.4314.

[0161] II. Focus on stability: LFTSAD, DCdetector and MSCRED show relatively stable performance in four indicators. For example, on the SMAP dataset, LFTSAD shows an accuracy of 0.9590 and an ACC of 0.9000, while USAD and IForest show very large fluctuations, especially the ACC of IForest changes significantly.

[0162] III. Considering anomaly scoring schemes: LFTSAD, DCdetector, and AnomalyTrans outperform models based on other schemes. For example, on the SMAP dataset, LFTSAD has an ACC of 0.9000, DTAAD has an ACC of 0.8874, and USAD has an ACC of 0.8134.

[0163] IV. Comparative Model Architecture: The shallow LFTSAD model surpasses several deep models. For example, on the PSM dataset, LFTSAD achieves an accuracy of 0.9785 and an F-Score of 0.9848, outperforming deep models such as AnomalyTrans (accuracy of 0.9545 and F-Score of 0.9825) and TranAD (accuracy of 0.9784 and F-Score of 0.9737).

[0164] Table IV: Performance of multiple models on multivariate datasets

[0165]

[0166]

[0167] 3) Visualization of performance: The anomaly detection results of all models at a time stamp are visualized on two real datasets, MSL and MITDB. Figure 4 As shown in the figure, the black line represents the true data curve, and the red or blue line represents the corresponding 0 / 1 label, where the 1 value represents an abnormality and the 0 value represents a normality. Figure 4 In (a) and (b), Ground-truth represents the real data and the real label, and the other sub-graphs represent the predicted labels of different models. From the figure, we can draw two conclusions:

[0168] I. Focusing on prediction accuracy: LFTSAD, DCdetector, AnomalyTrans, and DTAAD are closer to the true labels than other models. For example, on the MSL dataset, these models can identify anomalies in the range of 550-750 timestamps very well, while other models will make a lot of errors.

[0169] II. Considering the false positive problem: LFTSAD, DCdetector, AnomalyTrans, and DTAAD produce fewer false positives than other models. For example, on the MITDB dataset, DTAAD produces one false positive, LFTSAD produces two false positives, and DCdetector produces three false positives, while other models have many false positives.

[0170] 4) Summary: Experimental results show that the LFTSAD algorithm has good anomaly detection performance. Specifically, the LFTSAD model always ranks first or second in the four indicators of multiple datasets, outperforming multiple deep models.

[0171] (4) Time-effect experiment

[0172] The effectiveness of 10 models is compared on the MSL dataset from two aspects: the number of model parameters and the processing time per epoch. In general, the timeliness and resource consumption of neural network-based anomaly detection models mainly depend on the number of training parameters and the processing time per epoch with the same Batch_SIZE. Although the total number of epochs trained also affects the speed of the model, it is excluded because the number of epochs in different models is set arbitrarily and cannot accurately reflect the actual time consumption of the model. Specifically, the larger the training parameters, the greater the consumption of computing (CPU or GPU) and storage resources, the longer the parameter update time, and the longer the training time. Similarly, if the processing time per epoch is long, the longer the model training time, the longer the detection time, and the worse the timeliness of the model.

[0173] Table 5 Timeliness analysis

[0174]

[0175]

[0176] Table 5 lists the timeliness of the 10 models.

[0177] I. Focus on the number of parameters: LFTSAD has the fewest parameters among all the evaluated models, with only 12.6K parameters, which is significantly lower than other models. For example, the AnomalyTrans model has 4741K parameters, MSCRED has 1237.3K parameters, and PatchAD has 1477.9K parameters. Therefore, LFTSAD requires less CPU / GPU computing and storage resources and is very fast. These quality features make it perform well even in resource-limited Industrial IoT scenarios.

[0178] II. Consider the processing time of each epoch: LFTSAD has excellent timeliness, and each epoch only takes 26.5 seconds. In contrast, each epoch of the deep model takes more time, ranging from several times to dozens of times that of the LFTSAD model. For example, MTGFLOW takes 105.2 seconds, AnomalyTrans takes 201.1 seconds, and PatchAD takes 905.8 seconds. It is worth noting that IForest is a zero-parameter time series anomaly detection model based on machine learning, and each epoch only takes 19.7 seconds.

[0179] In summary, LFTSAD has fewer parameters and the fastest training speed, and its timeliness exceeds that of most time series anomaly detection models.

[0180] (5) Deployability Experiment

[0181] This example experiments the deployment of ten models on a resource-limited Raspberry Pi 4b edge device.

[0182] The Raspberry Pi 4b is equipped with a 1.5GHz ARM Cortex-A72 processor and 2GB RAM. In this experiment, all models are first trained in the cloud and then deployed to the edge device. Figure 7 The figure shows the deployment results on Raspberry Pi 4b. From the figure, two results can be found.

[0183] I. Deployability: Eight models were successfully deployed, but PatchAD and AnomalyTrans models could not be deployed because they required too much memory. In contrast, the shallow model deployment process is simpler and easier.

[0184] II. Resource consumption: In the deployment run of 100 timestamps, the shallow model consumes much less processing time, CPU utilization, and RAM utilization than the deep model, ensuring that it can run smoothly on edge devices with limited resources.

[0185] III. Convenience: In the IIoT environment, production processes change frequently, resulting in dynamic fluctuations in time signals. Faced with dynamically changing IIoT time signals, all models must be regularly fine-tuned in the cloud and then updated to edge devices. In comparison, shallow models are easy to update and consume fewer resources. Deep models are troublesome to update and consume more resources.

[0186] The above implementation cases are only preferred embodiments of the present invention and are not intended to limit the present invention in any form. Although the present invention has been disclosed as above with preferred embodiments, they are not intended to limit the present invention. Therefore, any simple modification, equivalent changes and modifications made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A lightweight time series anomaly detection method combining point-level and sequence-level reconstruction associations, characterized in that: The anomaly detection method first uses two two-layer MLPs as a skeleton to construct a lightweight pure MLP dual-branch reconstruction network to learn the local correlation and global correlation of each timestamp, and uses the difference between the local correlation and the global correlation as the reconstruction correlation; then, the correlation reconstruction learning is simultaneously performed from two complementary perspectives of point level and sequence level; finally, an anomaly scoring mechanism based on reconstruction correlation is designed to combine the point-level reconstruction correlation and the sequence-level reconstruction correlation to identify explicit anomalies and implicit anomalies in time series.

2. The lightweight time series anomaly detection method of combining point-level and sequence-level reconstruction association according to claim 1 is characterized in that: The anomaly detection method specifically includes the following contents: Step S1, point-level reconstruction association learning: construct a lightweight first two-branch reconstruction network consisting of two two-layer MLPs, learn the local association and global association of each timestamp from a point-level perspective, and generate point-level reconstruction associations; Step S2, sequence-level reconstruction association learning: construct another lightweight pure MLP two-branch reconstruction network to complement the first two-branch reconstruction network, learn the local and global associations of each timestamp from a sequence-level perspective, and generate sequence-level reconstruction associations; Step S3, anomaly scoring based on reconstructed associations: Combine the point-level and sequence-level reconstructed associations to generate an anomaly score for each timestamp and decide whether the current timestamp is an anomaly.

3. The lightweight time series anomaly detection method combining point-level and sequence-level reconstruction association according to claim 2 is characterized in that: The step S1 specifically includes the following steps: S1-1, point-level sampling: First, multiple sequences of length WIN_SIZE are randomly extracted in parallel from the time series; second, each sequence is further divided into N1 non-overlapping segments, each segment contains N2 timestamps, and the product of N1 and N2 is equal to WIN_SIZE; then, the N1 segments are stacked to form a 2D matrix of size N1×N2; finally, the local neighbors and global neighbors of each timestamp are quickly generated from the 2D matrix; for a current timestamp in the i-th row and j-th column, the local neighbors are composed of other timestamps in the i-th row, and the global neighbors are other timestamps in the j-th column; S1-2, local association learning: a two-layer MLP network is used as the first branch of a lightweight two-branch reconstruction network to learn the association between each timestamp and its local neighbors; S1-3, global association learning: another two-layer MLP is used as the second branch of the lightweight two-branch reconstruction network to learn the association of each timestamp with its global neighbors; S1-4, loss function: The local association learning process and the global association learning process constitute two contrast branches, and the MSE function is used as the loss function of contrast learning to align the reconstruction values ​​of the two branches.

4. The lightweight time series anomaly detection method of combining point-level and sequence-level reconstruction association according to claim 3 is characterized in that: In S1-2 and S1-3, the two-layer MLP network includes an input layer with N2-1 neurons and a hidden layer with d neurons, the default value of the hidden layer is 64 or 128, and an output layer with 1 neuron; the input layer and the hidden layer constitute a first fully connected network with (N2-1)×d trainable parameters; the hidden layer and the output layer form a second fully connected network with d×1 trainable parameters; the hidden layer uses dropout and ReLU activation functions.

5. The lightweight time series anomaly detection method combining point-level and sequence-level reconstruction association according to claim 3 is characterized in that: The step S2 specifically includes the following steps: S2-1, sequence-level sampling: First, multiple sequences of length W_SEQ are randomly and parallelly extracted from the time series; then, each sequence is divided into N3 non-overlapping segments; at the same time, each segment is divided into N4 non-overlapping sub-segments, each sub-segment contains N5 timestamps, and the product of N3, N4 and N5 is equal to W_SEQ; then, the N3 segments are stacked to form a 2D matrix of size N3×(N4×N5); finally, the sequence-level local neighbors and global neighbors of each timestamp are generated from the 2D matrix. For a current timestamp located in the i-th row and the j-th column, the sub-segment of the sequence to which it belongs is used as the replacement sequence, the other sub-segments in the row i are used as local neighbors, and the sub-segments corresponding to other timestamps in the column j are used as global neighbors; S2-2, local association learning: a two-layer MLP network is used as the first reconstruction branch to learn the association between each timestamp and its local neighbors from a sequence perspective; S2-3, global association learning: another two-layer MLP network is used as the second reconstruction branch to learn the association between each timestamp and its global neighbors from a sequence perspective; S2-4, loss function: The local association learning process and the global association learning process constitute two contrast branches, and the MSE function is used as the contrast learning loss function.

6. The lightweight time series anomaly detection method of combining point-level and sequence-level reconstruction association according to claim 5 is characterized in that: In S2-2 and S2-3, the two-layer MLP network includes an input layer with N4-1 neurons and a hidden layer with d neurons, the default value of the hidden layer is 64 or 128, and an output layer with N5 neurons; the input layer and the hidden layer constitute a first fully connected network; the hidden layer and the output layer form a second fully connected network; the hidden layer uses dropout and ReLU activation functions, and the two-layer MLP network contains N4×N5×d training parameters.

7. The lightweight time series anomaly detection method of combining point-level and sequence-level reconstruction association according to claim 6 is characterized in that: In the step S3, specifically: S3-1, set a timestamp x in the test set t Input into point-level and sequence-level reconstruction association learning at the same time to perform parallel learning; S3-2, parallelizing the correlation calculation of M variables at the t-th timestamp in point-level reconstruction correlation learning and sequence-level reconstruction correlation learning; After the calculation is completed, M reconstructed association values ​​of the t-th timestamp are generated respectively; S3-3, calculate the average value of the M reconstruction association values ​​in each reconstruction association learning, and generate the point-level reconstruction association of timestamp t and sequence-level reconstruction S3-4, the anomaly score of the t-th timestamp is generated by combining the point-level reconstruction association and the sequence-level reconstruction association, which is: Score(x t )=α×Score P (x t )+(1-α)×Score S (x t ) (13) Among them, α is a preset parameter, ranging from [0 to 1]; when α = 0, only the sequence-level reconstruction association is considered; when α = 1, only the point-level reconstruction association is considered; represents the point-level reconstruction association on the mth variable, represents the sequence-level reconstruction association on the mth variable; S3-5, after obtaining the anomaly score, determine whether the tth timestamp is an anomaly using formula (2): Where λ is a preset threshold ranging from 0 to 1; if Indicates that the tth timestamp is abnormal; otherwise, the tth timestamp is normal.

Images