Anomaly detection method and system for IoT time series data based on dynamic graph attention

Abstract

The present invention discloses an anomaly detection method and system for IoT time series data based on dynamic graph attention, which relates to the field of computing technology. The method comprises S1 constructing an initial anomaly detection model; S2 obtaining a training data set; S3 using the training data set to train and optimize the initial anomaly detection model to obtain an optimized anomaly detection model; S4 obtaining the IoT time series data to be predicted; S5 using the optimized anomaly detection model to analyze the detection results of the IoT time series data to be predicted; the method solves the problem of multivariate time series anomaly detection in the IoT environment; overcomes the traditional Transformers and GNN defects, a method with exceptional attention is proposed Transformers Get GNN The effective modulus of the parameters; this anomaly detection model has achieved advanced results in empirical studies and improved the accuracy of anomaly detection.

Description

Technical Field

[0001] The present invention relates to the field of computing technology, and in particular to a method and system for detecting anomaly in time series data of the Internet of Things based on dynamic graph attention. Background Art

[0002] With the rapid growth of interconnected devices and sensors in cyber-physical systems such as self-driving cars, smart buildings, water treatment and distribution plants, the emergence of the Internet of Things has further promoted the application of cyber-physical systems in various tasks. There is an increasing need to monitor these devices from attacks, which is particularly important for critical infrastructure such as power grids and communication networks. The Internet of Things can use blockchain, edge computing, deep learning and other methods to achieve target monitoring, positioning, identification, user privacy and security storage and other functions. Nowadays, the amount of data is growing exponentially. In the face of these massive amounts of data, the improvement of the Internet of Things can greatly promote the future application and development of wireless sensor networks. Therefore, it is very necessary to perform anomaly detection on the monitored data, that is, to analyze the abnormal behavior of the monitored object by analyzing the abnormal values ​​of the target monitoring data. How to accurately and efficiently perform anomaly detection has also become a hot issue in the field of Internet of Things security.

[0003] Anomaly detection has a wide range of applications in many fields, including network security, medicine, machine vision, statistics, credit card fraud, and large-scale expenditures. Anomaly detection algorithms mainly learn to detect anomalies or send out danger signals when abnormal events occur by observing unlabeled data sets of normal events. Due to the lack of abnormal labels in the data collected by sensors, and the fact that anomalies are often unpredictable and diverse, anomaly detection is usually regarded as an unsupervised learning problem. The vast majority of data in the Internet of Things exists in the form of multivariate time series. Time dependence is a very important feature of IoT data, which has a great impact on data prediction and analysis. Therefore, the essence of IoT anomaly detection is time series anomaly detection. Based on this, we will focus on time series anomaly detection in an unsupervised environment.

[0004] Before machine learning, traditional time series algorithms were generally used to build statistical models, and some results were achieved. However, the multivariate time series generated by today's cyber-physical systems are highly complex, and the inherent correlation is nonlinear. These methods simply model the relationship between sensors and can only capture linear relationships. Therefore, in recent years, researchers have begun to use deep learning-based technologies to detect anomalies in high-dimensional data, thereby developing more intelligent and cost-effective methods to identify anomalies. For example, unsupervised anomaly detection algorithms such as single-class support vector machine OCSVM, ridge regression, recurrent neural network RNN, and long short-term memory LSTM are used to build models. These deep learning methods can capture long-term dependencies in time series data and are suitable for processing abnormal data with temporal characteristics. However, due to the relatively complex internal structure, the training efficiency is very low, the amount of calculation will be large, and it is very time-consuming. In addition, the distribution assumptions for abnormal data are weak, which is easily affected by normal data, which may lead to false detection and missed detection. In particular, the training accuracy of these models is not high enough, so there is still room for improvement in optimizing model performance. In recent years, with the further development of deep learning technology, attention Transformer and graph neural network GNN have achieved a series of important results in many fields.

[0005] Nowadays, new applications of Transformer self-attention networks have been recognized, published, and successfully used in research fields such as computer vision, image processing, and natural language processing. Its structure includes modules such as self-attention mechanism, position encoding, normalization Add&Normalize, and fully connected layer Feed Forward. These modules work together to achieve fast parallel operations by using self-attention mechanism, and can better process sequence data and extract temporal features. Graph neural network GNN can consider both spatial and temporal dimensions of data by learning the temporal representation of graph structure, and has stronger ability in modeling spatiotemporal features. Therefore, we consider fusing attention transformer with graph neural network GNN to complete the task of time series anomaly detection. In recent years, graph neural network GNN has been widely studied and successfully completed difficult machine learning tasks such as node classification, link prediction, and graph classification, thanks to the high expressiveness achieved through message passing in effective learning graph representation. Graph neural network GNN can learn temporal and spatial dependencies, show high-dimensional data with complex relationships, and can be widely used in modeling complex systems. Based on this, graph neural network GNN is a promising way to model multivariate time series data. It takes graph-structured data as input. If it is applied to anomaly detection of multivariate time series, the complex relationships in the time series need to be converted into graphs and learned together with the model.

[0006] Research on time series anomaly detection has been going on for decades and is an active research field that is gaining more and more attention in machine learning and data mining. At the same time, many models for time series anomaly detection have been proposed. The current research status of existing statistical-based methods and deep learning-based methods is as follows:

[0007] 1. Traditional statistical methods are mainly used on single-feature time series data, and most of them are linear methods. For example, the differential integrated moving average autoregressive ARIMA model is used to predict time series data. Chen et al. proposed the use of isolation forests and elliptical envelopes for geochemical anomaly detection. These methods are relatively effective for short-term linear time series data prediction, but the effect is not ideal for long-term time series data prediction. As time series data becomes more and more dimensional and complex, these methods can no longer meet current needs. Due to the powerful representation ability of deep neural networks, deep learning methods have received widespread attention.

[0008] 2. Currently, two popular deep learning models, convolutional neural network (CNN) and recurrent neural network (RNN), are widely used for anomaly detection. These models usually use long short-term memory (LSTM) layers and stacked convolutional neural network (CNN) layers to extract features from time series, and then apply an exponentially normalized softmax layer to predict labels. In order to predict more accurately, a long short-term memory variational autoencoder (LSTM-VAE) model has been proposed, which uses a long short-term memory (LSTM) backbone for temporal modeling and a variational autoencoder (VAE) for reconstruction. The multivariate long short-term memory fully convolutional network (MLSTM-FCN) uses a long short-term memory (LSTM) layer and a stacked convolutional neural network (CNN) layer along the squeeze and excitation block to generate latent features. The task-adaptive mapping network (TapNet) also constructs a long short-term memory (LSTM) layer and a stacked convolutional neural network (CNN) layer. The bidirectional recurrent neural network (BiRNN) model improves the prediction accuracy of the model by adding a direction to the general recurrent neural network (RNN). However, the limitations of the above models are obvious: they assume that time series variables have the same influence, so they cannot explicitly model the pairwise dependencies between variables, and the model accuracy is not high enough.

[0009] In traditional anomaly detection, attention transformers can be used to capture global dependencies and contextual information of data. By introducing the self-attention mechanism in the attention transformer, the model can pay more attention to important nodes and edges, thereby capturing abnormal patterns and features. Graph neural networks can be used to model relationships and dependencies between data. Both have their own advantages. Graph attention transformers provide an example of how to generalize the attention transformer architecture to graphs. This method introduces the topological structure properties of graphs into the attention transformer, giving the model a structural position prior in high-dimensional space. Using Laplacian eigenvectors as absolute encodings and calculating attention on the immediate area of ​​each node instead of the entire graph, it combines the core of the attention transformer (global attention) and the core of the graph neural network GNN (considering the topological properties of the graph). The storage area network SAN is similar to it, but it calculates attention on the entire graph, while distinguishing between real edges and created edges. Mialon et al. proposed a relative encoding through the kernel on the graph to bias the self-attention calculation, and then the position information can be incorporated into the attention transformer by selecting the kernel function. All of them explicitly incorporate graph structures to design a graph attention Transformer architecture that takes into account both local and global information.

[0010] Due to the huge amount of time series data generated in the IoT environment and the complex dimensional relationship characteristics, the existing anomaly detection methods still have the following shortcomings: ① The prediction accuracy is still not high enough. ② Past methods are to statically mine the features in the time series, while ignoring that the time series is dynamically evolving over time. Summary of the invention

[0011] The purpose of the present invention is to design a method and system for detecting anomalies in IoT time series data based on dynamic graph attention in order to solve the above problems.

[0012] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0013] The IoT time series data anomaly detection method based on dynamic graph attention includes:

[0014] S1. Construct an initial anomaly detection model. The anomaly detection model includes an input layer, a linear branch, a nonlinear branch and an output layer. The output of the input layer is used as the input of the linear branch and the nonlinear branch. The output of the linear branch and the nonlinear branch are used as the input of the output layer. The output layer is used to analyze the final prediction result according to the output of the linear branch and the output of the nonlinear branch. The nonlinear branch includes a feature extraction module with anomaly attention and a dynamic graph neural network module from input to output. The feature extraction module is used to extract the time feature matrix of the input data. The dynamic graph neural network module is used to dynamically model and predict the time feature matrix to obtain a nonlinear prediction result.

[0015] S2, obtain training data set;

[0016] S3, using the training data set to train and optimize the initial anomaly detection model to obtain an optimized anomaly detection model;

[0017] S4, obtaining the IoT time series data to be predicted;

[0018] S5. Use the optimized anomaly detection model to analyze the detection results of the IoT time series data to be predicted.

[0019] The IoT time series data anomaly detection system based on dynamic graph attention includes:

[0020] Storage; Storage is used to store programs;

[0021] Executor; the executor is used to execute the program stored in the storage. When the executor executes the program stored in the storage, the above-mentioned method for detecting anomalies in IoT time series data based on dynamic graph attention is implemented.

[0022] The beneficial effects of the present invention are as follows: the present method solves the problem of anomaly detection in multivariate time series in the IoT environment; overcomes the defects of traditional attention Transformer and graph neural network GNN, and proposes an effective model of attention Transformer and graph neural network GNN parameters with anomaly attention; the present anomaly detection model has achieved advanced results in empirical studies and improved the accuracy of anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is the structural diagram of this model;

[0024] Figure 2 for Figure 1 Enlarged view of (1)

[0025] Figure 3 for Figure 1 Enlarged view of (2) in the middle;

[0026] Figure 4 for Figure 1 Enlarged view of (3);

[0027] Figure 5 This is the flowchart of anomaly detection for this model;

[0028] Figure 6 A comprehensive ranking diagram of the average F1 scores of the model and some baseline methods on three datasets;

[0029] Figure 7 It is a dimensional feature representation diagram of the dataset MSL;

[0030] Figure 8 It is a feature representation diagram of one dimension of the dataset SMAP;

[0031] Fig. 9 It is a feature representation diagram of one dimension of the dataset PSM;

[0032] Fig.10 is the statistical information of the data set;

[0033] Fig.11 The detection performance of this method is compared with 8 baseline models on 3 datasets;

[0034] Fig.12 This is the ablation experiment comparison of each module of this method on three datasets. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0036] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0038] The IoT time series data anomaly detection method based on dynamic graph attention includes:

[0039] S1. Construct an initial anomaly detection model. The anomaly detection model includes an input layer, a linear branch, a nonlinear branch and an output layer. The output of the input layer is used as the input of the linear branch and the nonlinear branch. The output of the linear branch and the nonlinear branch are used as the input of the output layer. The output layer is used to analyze the final prediction result according to the output of the linear branch and the output of the nonlinear branch. The nonlinear branch includes a feature extraction module with abnormal attention and a dynamic graph neural network module from input to output. The feature extraction module is used to extract the time feature matrix of the input data. The dynamic graph neural network module is used to dynamically model and predict the time feature matrix to obtain a nonlinear prediction result. The linear branch is an autoregressive model.

[0040] The feature extraction module includes M feature extraction networks connected in sequence from input to output, each feature extraction network includes an abnormal attention layer, a first normalization layer, a feedforward layer and a second normalization layer, the abnormal attention layer adopts a multi-head attention mechanism with H heads, the multi-head attention mechanism is used to model the prior association and sequence association of each time point, the output of the input layer is used as the input of the abnormal attention layer, the output of the abnormal attention layer and the output of the input layer are both used as the input of the first normalization layer, the output of the first normalization layer is used as the input of the feedforward layer, the output of the feedforward layer and the output of the first normalization layer are both used as the input of the second normalization layer, and M and H are both positive integers greater than 1;

[0041] The abnormal attention layer includes a prior association branch and a sequence association branch. The prior association branch uses a learnable Gaussian kernel to calculate the prior of relative time distance. The Gaussian kernel uses a learnable scale parameter , so that the prior association adapts to various time series patterns. The sequence association branch is used to learn the association relationship from the original sequence. The sequence association branch can adaptively find the most effective association relationship. The maximum and minimum strategies are adopted between the prior association branch and the sequence association branch. The minimization stage of the maximum and minimum strategies is to drive a prior association to simulate the sequence association learned from the original sequence; the maximization stage of the maximum and minimum strategies is to optimize the sequence association and expand the association difference; this stage aims to extract time features and construct feature matrices And get abnormal attention. The feature matrix of each sequence is expressed as , where Embed1 represents the abnormal attention function, X∈{x1, x2,…, x n}, is a time-ordered sequence of measurements observed by N sensors in coordination with each other, x t represents the observed value at time t;

[0042] The prior association branch of the abnormal attention layer calculates the prior with respect to the relative time distance, and then projects the input node feature X to the query Q, key K and value V matrix through linear projection. Suppose there are M feature extraction networks with a layer length of n and an input time series , then the abnormal attention in the mth layer is expressed as:

[0043] ;

[0044] ;

[0045] in, According to the learning scale Generate the m-th layer of prior correlation , , the i-th element Corresponding to time point i th , Used to convert the relevant weights into a discrete distribution P by dividing the rows m ; the i-th element The association weight with the jth point is given by the Gaussian kernel , where the exp() function represents an exponential function with the natural constant e as the base, and the distance |ji| is calculated; represents a sequence association, represents the normalization of attention along the last dimension, that is, N of the current layer model The number of normalizations is ; the learning scale of the abnormal attention layer is , Represents the query, key, and value of the hth head, respectively, and concatenates the outputs of multiple heads And get the final result , the anomaly difference adopts the symmetric KL difference of prior association and serial association to express the information gain between the two distributions, which is expressed as:

[0046] ;

[0047] Among them, KL (·||·) is the m and Z m The KL divergence is calculated between two discrete distributions for each row of , In other words, KL divergence calculates the expected value of the logarithmic difference between the original distribution of the data and the probability of the approximate distribution; is the pointwise correlation difference of X relative to the prior correlation P and sequence correlation Z from multiple layers;

[0048] The dynamic graph neural network module includes a graph structure learning layer and multiple neural network blocks connected in sequence from input to output. The graph structure learning layer is used to capture the spatiotemporal features of the time series and adaptively generate an adjacency matrix. Each neural network block includes a graph neural network GNN and a pooling layer from input to output. The graph neural network GNN is used to model the spatiotemporal relationship, and the pooling layer is used to perform soft clustering on the graph neural network GNN.

[0049] The dynamic graph neural network module dynamically models and predicts the time feature matrix to obtain nonlinear prediction results. Specifically, it uses time windows to process temporally continuous data, and applies graph neural network GNN to it in each time window for feature learning, and then processes the data of the entire time series by sliding the time window to capture its dynamic evolution process. The graph structure learning layer adaptively learns the adjacency matrix of the Transformer sequence with anomaly attention feature extraction module to simulate the potential relationship in a given time series sample. The adjacency matrix A learned by the graph structure learning layer (i) It is expressed as:

[0050] ;

[0051] Among them, Embed2 represents the graph structure learning function;

[0052] Compute the similarity matrix between sampled time series , expressed as:

[0053] ;

[0054] Where distance represents the distance degree, such as Euclidean distance, absolute value distance, dynamic time warping, etc.; then the dynamic adjacency matrix A (i) It can be expressed as:

[0055] ;

[0056] Where W1 is the learnable model parameter and f is the activation function;

[0057] Set the threshold b to make the adjacency matrix sparse, expressed as:

[0058] ;

[0059] A (i) Apply normalization;

[0060] Then, three graph neural networks (GNNs) (G1, G2, and G3) were used to train the input graph (denoted as X (i) , A (i)) is used to model spatiotemporal relationships. The graph neural network GNN fuses spatial dependencies and temporal patterns to embed the features of nodes, and converts the feature dimensions of the nodes into decoding, that is, into spatial dependencies and temporal patterns, expressed as:

[0061] ;

[0062] in Represents the learned node embedding from spatiotemporal modeling. G1(), G2(), and G3() are composed of a graph neural network layer GNN and a batch normalization layer. The graph neural network layer GNN can be a graph convolutional network GCN, a graph attention network GAT, a graph isomorphism network GIN, etc.

[0063] The pooling layer uses the diffusion pool Diffpool to train the graph neural network layer GNN and learn the soft clustering allocation of nodes in each layer of the deep graph neural network layer GNN. The process of the pooling layer is expressed as:

[0064] ;

[0065] ;

[0066] in, , , is a trainable parameter matrix representing a linear transformation, and is the assignment matrix representing the projection from the original node to the pooled node, (S (i) ) T Denotes the allocation matrix S (i) The inverse matrix of the distribution matrix S (i) To generate centroids based on the input graph , and then calculate the relationship between the centroid of each batch of aggregations and the encoding graph;

[0067] Use cosine similarity to evaluate input node embeddings and centroid The relationship between , expressed as:

[0068] ;

[0069] Among them, cosine similarity () is also called cosine similarity, which is calculated by and The cosine of the angle between them is used to evaluate their similarity. It is equal to and The dot product of and The product of the lengths (or sizes).

[0070] Row normalization is applied in the resulting distribution matrix, expressed as:

[0071] ;

[0072] The row normalization () is also called norm, which scales the feature vector of each sample to the unit norm, that is, adjusts the feature vector of each sample so that their Euclidean distance is 1.

[0073] We cascade the matrix ,get , which is to combine multiple Concatenate along a specified axis and perform a trainable weighted sum of the concatenated matrices , and get the final allocation matrix S (i) ;

[0074] ;

[0075] After stacking several diffusion pools, the original graph is pooled into a single node and a graph-level representation vector is obtained. , which is the result of nonlinear branch training, expressed as:

[0076] .

[0077] Among them, P1(), P2(), and P3() represent three diffusion pooling Diffpool layers, and the encoded graph is hierarchically pooled to a single node for generating its graph-level representation. After stacking three pooling layers, we get its graph-level representation vector. In the linear branch, an autoregressive linear model is used to extract the linear part of the dataset to supplement the overall performance outside the nonlinear branch. A linear and nonlinear mixture is used as the final result to enhance the recognition ability of linear features.

[0078] S2. Get the training data set, which is a set of time series , which is the measurement value observed in time sequence by N sensors working together. Usually the time interval between two consecutive measurements is constant, where represents the observed value at time t;

[0079] S3. Using the training data set to train and optimize the initial anomaly detection model to obtain an optimized anomaly detection model; training and optimizing the initial anomaly detection model specifically includes:

[0080] ①. The nonlinear data of the training data set is passed into the nonlinear branch of the current anomaly detection model and enters ②. The linear data of the training data set is passed into the linear branch of the current anomaly detection model and enters ⑤.

[0081] ② The nonlinear branch feature extraction module models the prior correlation and sequence correlation of each time point, calculates the prior correlation and sequence correlation, obtains abnormal attention through the feature extraction module and constructs the time feature matrix;

[0082] ③ The graph structure learning layer dynamically captures the spatiotemporal characteristics of the time series and adaptively generates the adjacency matrix;

[0083] ④. The node embedding vectors of the pooling layer perform soft clustering on the graph neural network GNN, establish B neural network blocks through repeated superposition, and output the results of nonlinear branch training ;

[0084] ⑤. Predict future data by establishing a regression equation and then output the results of linear branch training ;

[0085] ⑥Weighted integration of the results of the nonlinear branch and the results of the linear branch training is performed to calculate the loss of the current anomaly detection model;

[0086] ⑦. Update the model parameters of the current anomaly detection model according to the loss, and record it as the next anomaly detection model;

[0087] ⑧. Determine whether the preset number of training times is completed. If so, output the next anomaly detection model as the optimized anomaly detection model. Otherwise, make the next anomaly detection model the current anomaly detection model and return to ①.

[0088] S4, obtaining the IoT time series data to be predicted;

[0089] S5. Use the optimized anomaly detection model to analyze the detection results of the IoT time series data to be predicted; specifically:

[0090] The result of the nonlinear branch of the anomaly detection model is , the result of linear branching is ;

[0091] The results of the nonlinear branch and the linear branch are weighted to obtain the final result. , expressed as:

[0092] ;

[0093] in, is element-wise multiplication;

[0094] According to the final result Compared with the pre-set threshold, the final result If it is greater than the threshold, the output detection result is an abnormal value, otherwise it is a normal value.

[0095] The IoT time series data anomaly detection system based on dynamic graph attention includes:

[0096] Storage; Storage is used to store programs;

[0097] Executor; the executor is used to execute the program stored in the storage. When the executor executes the program stored in the storage, the above-mentioned method for detecting anomalies in IoT time series data based on dynamic graph attention is implemented.

[0098] experiment

[0099] 1. Dataset

[0100] In the experiments, real datasets related to the Internet of Things are mainly selected to evaluate the models in this method: first, two public datasets of the Internet of Things, MSL (Mars Science Laboratory Rover) and SMAP (Soil Moisture Active and Passive Satellite), contain remote sensing anomaly data obtained in the spacecraft monitoring system event unexpected anomaly emergency anomaly (ISA) report. The dataset PSM (Pool Server Metrics) collected in multiple application server nodes is also used as a supplementary dataset. Fig.10 Statistics for these datasets are reported. Figure 7 , Figure 8 , Fig. 9 The one-dimensional feature representations of the three datasets are described respectively. It can be seen that there are significant differences in the feature distributions between them, which also shows that the selected datasets have diverse distributions.

[0101] The following comparative experiments are used to compare the performance of this method with existing methods.

[0102] 2. Model implementation process

[0103] (1) The following is the model training process

[0104] 1) Figure 1 , Figure 2 , Figure 3 Section A in the figure is the feature extraction module Transformer with an abnormal attention mechanism during nonlinear branch training. The main purpose of designing this module is to extract temporal features based on key observations of association differences, and to model both prior associations and sequence associations, reflecting association differences and obtaining abnormal attention. The overall structural feature of the feature extraction module Transformer is that abnormal attention layers and feedforward layers are alternately superimposed. This superposition structure is conducive to learning deep associations from deep multi-level features, thereby accelerating training and improving generalization performance. The specific steps are as follows: Figure 5As shown in the figure, the number of training times is set first during model training, and the time series data observed by multiple sensors working together is input into the anomaly detection model. Since the input data contains both linear and nonlinear structures, a parallel structure model is designed to better process these data, and the nonlinear features in the data are trained in the nonlinear branch. First, the input is Figure 1 The A section in the training is trained to model the prior association and serial association of each time point respectively, and calculate the prior association and serial association. In addition, the maximum and minimum strategies are adopted between the two branches to enhance the normal-abnormal distinguishability of the association difference. Then, through the alternating superposition of the abnormal attention layer and the feedforward layer, the abnormal attention is obtained and the time feature matrix is ​​constructed.

[0105] 2) Figure 1 , Figure 4 The B, C and D blocks in the middle are the dynamic graph neural network modules of the anomaly detection model during nonlinear branch training. The main purpose of the dynamic graph neural network module is to dynamically model and predict the spatiotemporal information of the observations, and finally obtain the final result of the graph-level representation vector output nonlinear module. The specific steps are as follows Figure 5 As shown in the figure, the spatiotemporal features of the time series are first captured dynamically through the graph structure learning layer, and the adjacency matrix is ​​adaptively generated. Next, the node embedding vectors of the diffusion pooling layer are used to softly cluster the GNN layers, and a deep GNN is established through repeated superposition, and then the results of nonlinear branch training are output.

[0106] 3) Figure 1 E in the figure is the autoregressive module of the anomaly detection model during linear branch training. The purpose of the autoregressive module is to extract the linear part of the observation value to enhance the recognition ability of linear features. The specific steps are as follows: Figure 5 As shown in Figure 1, the linear features in the data are fed into the autoregressive model for training. The regression equation is established to predict future data and then the results of the linear training module are output.

[0107] 4) The anomaly detection model obtains the nonlinear branch training results and the linear branch training results by processing the time series data in parallel, and then weights and integrates the results of these two parts to calculate the loss of the anomaly detection model. Then, the updated model parameters are used to continue to update the parameters of subsequent model training, and the model is trained in the same way until the training times are completed.

[0108] (2) The following is the model testing process

[0109] 1) Input the test data into the trained nonlinear branch and the trained linear branch respectively for anomaly detection and output their anomaly scores.

[0110] 2) Specific steps are as follows Figure 5 As shown in the figure, first, the test data is input into the trained feature extraction module Transformer with an abnormal attention mechanism, and the prior loss and sequence loss are calculated through the prior association and sequence association, and then the abnormal loss of this part is calculated. Then, the adjacency matrix is ​​generated through the graph structure learning layer and input into the trained neural network block to obtain the abnormal score of the nonlinear branch. At the same time, the test value is also input into the trained autoregressive model to obtain the abnormal score of the linear branch, and then the two parts of the abnormal score are added according to the weights.

[0111] 3) When detecting anomalies, an anomaly threshold is set, and the obtained anomaly score is compared with the anomaly threshold to obtain the test result of the anomaly detection model. When the anomaly score exceeds the threshold, the point is considered an anomaly point, otherwise it is a normal point, and the test result is finally output. In order to improve the detection performance of the anomaly detection model, softmax is used to obtain the final anomaly score to improve the detection accuracy in anomaly detection. The larger the final result value, the better the ability to detect anomalies.

[0112] 3. Model performance indicators

[0113] The performance of the models is compared using several main performance metrics based on the confusion matrix: precision, recall, and F1 score. For these three metrics, higher values ​​indicate better performance.

[0114] Precision Pre refers to the proportion of samples predicted correctly in the prediction of positive examples based on the prediction results of the model. There are two types of results predicted as positive examples, either actually positive examples TP or actually negative examples FP, which can be expressed as:

[0115] ;

[0116] Recall rate Rec refers to the proportion of positive samples predicted correctly to the total positive samples among the samples that are actually positive examples. Among the samples that are actually positive examples, either they are predicted correctly in the prediction TP or they are predicted incorrectly in the prediction FN. It can be expressed as:

[0117] ;

[0118] The F1 score is the harmonic mean of precision and recall, and is calculated as:

[0119] ;

[0120] Precision, recall and F1 score refer to the arithmetic mean of each statistical indicator value across all categories.

[0121] Model comparison results

[0122] from Figure 6 , Fig.11 and Fig.12 It can be seen that compared with the existing models, the experimental results of this model on three real data sets are as follows:

[0123] from Fig.11 It can be seen that our model achieves consistent state-of-the-art performance in all baseline model tests.

[0124] The F1 scores of this model on the MSL, PSM, and SMAP datasets are 94.92%, 98.11%, and 96.53%, respectively, which are 20.34%, 18.2%, and 19.48% higher than the average F1 scores of other methods. We observe that the advantages of considering the fusion of attention Transformer and graph neural network GNN in time series anomaly detection are convincing.

[0125] from Fig.12 It can be seen that the impact of using dynamic graph embedding strategy on anomaly detection.

[0126] Dynamic Graph Attention DGFormer-one with all-ones adjacency matrix is ​​a dynamic graph attention DGFormer framework with all-ones adjacency matrix. Dynamic Graph Attention DGFormer-corr with Correlation Coefficient Adjacency Matrix is ​​a dynamic graph attention DGFormer framework with Correlation Coefficient Adjacency Matrix. The dynamic graph attention DGFormer framework has the dynamic adjacency matrix proposed by the present invention. It can be seen that different adjacency matrices can be used in this model, however, the performance of the all-ones matrix is ​​slightly worse than that of the correlation coefficient matrix, and the dynamic matrix of the present invention achieves the best.

[0127] from Fig.12 It can also be seen that the hybrid attention Transformer and graph neural network GNN have a higher average F1 score than other combinations.

[0128] Dynamic Graph Attention DGFormer-woAR without Autoregression means that the dynamic graph attention DGFormer model removes the autoregressive component, and dynamic graph attention DGFormer-woDG without dynamic graph means that the dynamic graph attention DGFormer model removes the dynamic graph embedding and graph neural network components. The complete dynamic graph attention DGFormer obtains the best results on different batch sizes, indicating that all components contribute to the effectiveness and accuracy of the entire model. The performance of dynamic graph attention DGformer-woDG without dynamic graph has declined, indicating that adding graph neural network to dynamically capture temporal features can improve the timeliness of the model. The performance of dynamic graph attention DGformer-woAR without autoregression has declined more significantly, indicating that the autoregressive component plays a vital role.

[0129] Figure 6 The average F1 score of this model and some baseline methods is shown. As can be seen from the figure, this model has the highest F1 score on all three datasets, indicating that our model can better detect outliers for time series data, making it have higher performance in anomaly detection.

[0130] The technical solution of the present invention is not limited to the above-mentioned specific embodiments. All technical variations made according to the technical solution of the present invention fall within the protection scope of the present invention.

Claims

1. Anomaly detection method for IoT time series data based on dynamic graph attention, characterized in that: include: S1. Construct an initial anomaly detection model. The anomaly detection model includes an input layer, a linear branch, a nonlinear branch and an output layer. The output of the input layer is used as the input of the linear branch and the nonlinear branch. The output of the linear branch and the nonlinear branch are used as the input of the output layer. The output layer is used to analyze the final prediction result according to the output of the linear branch and the output of the nonlinear branch. The nonlinear branch includes a feature extraction module with anomaly attention and a dynamic graph neural network module from input to output. The feature extraction module is used to extract the time feature matrix of the input data. The dynamic graph neural network module is used to dynamically model and predict the time feature matrix to obtain a nonlinear prediction result. S2, obtain training data set; S3, using the training data set to train and optimize the initial anomaly detection model to obtain an optimized anomaly detection model; Training and optimizing the initial anomaly detection model specifically includes: ①. The nonlinear data of the training data set is passed into the nonlinear branch of the current anomaly detection model and enters ②. The linear data of the training data set is passed into the linear branch of the current anomaly detection model and enters ⑤. ② The nonlinear branch feature extraction module models the prior correlation and sequence correlation of each time point, calculates the prior correlation and sequence correlation, obtains abnormal attention through the feature extraction module and constructs the time feature matrix; ③ The graph structure learning layer dynamically captures the spatiotemporal characteristics of the time series and adaptively generates the adjacency matrix; ④. The node embedding vectors of the pooling layer perform soft clustering on the graph neural network GNN, establish B neural network blocks by repeated superposition, and output the results of nonlinear branch training B is a positive integer greater than 1; ⑤. Predict future data by establishing a regression equation and then output the results of linear branch training ⑥Weighted integration of the results of the nonlinear branch and the results of the linear branch training is performed to calculate the loss of the current anomaly detection model; ⑦. Update the model parameters of the current anomaly detection model according to the loss, and record it as the next anomaly detection model; ⑧. Determine whether the preset number of training times is completed. If so, output the next anomaly detection model as the optimized anomaly detection model. Otherwise, make the next anomaly detection model the current anomaly detection model and return to ①; S4. Obtain the IoT time series data to be predicted; S5. Use the optimized anomaly detection model to analyze the detection results of the IoT time series data to be predicted.

2. The method for detecting anomaly in IoT time series data based on dynamic graph attention according to claim 1 is characterized in that: The feature extraction module includes M feature extraction networks connected in sequence from input to output, each feature extraction network includes an abnormal attention layer, a first normalization layer, a feedforward layer and a second normalization layer. The abnormal attention layer adopts a multi-head attention mechanism with H heads. The multi-head attention mechanism is used to model the prior association and sequence association of each time point. The output of the input layer is used as the input of the abnormal attention layer, the output of the abnormal attention layer and the output of the input layer are both used as the input of the first normalization layer, the output of the first normalization layer is used as the input of the feedforward layer, the output of the feedforward layer and the output of the first normalization layer are both used as the input of the second normalization layer, and M and H are both positive integers greater than 1.

3. The method for detecting anomaly in IoT time series data based on dynamic graph attention according to claim 2 is characterized in that: The abnormal attention layer includes a prior association branch and a sequence association branch. The prior association branch uses a Gaussian kernel to calculate the prior of relative time distance. The sequence association branch is used to learn the association relationship from the original sequence. The maximum and minimum strategies are adopted between the prior association branch and the sequence association branch. The minimization stage of the maximum and minimum strategies is to drive a prior association to simulate the sequence association learned from the original sequence; the maximization stage of the maximum and minimum strategies is to optimize the sequence association and expand the association difference.

4. The method for detecting anomaly in IoT time series data based on dynamic graph attention according to claim 3 is characterized in that: The prior association branch of the abnormal attention layer calculates the prior of the relative time distance, and then projects the input node feature X to the query Q, key K and value V matrix through linear projection. Suppose there are M feature extraction networks with a layer length of n and an input time series X∈R n×N , then the abnormal attention in the mth layer is expressed as: in, According to the learning scale σ∈R n×1 Generate the prior correlation P of the mth layer m ∈R n×n ,m∈{1,...,M}, the i-th element σ i Corresponding to time point i th , Re scale(·) is used to convert the relevant weights into a discrete distribution P by dividing the rows m ; the i-th element σ i The association weight with the jth point is given by the Gaussian kernel Determine, where the exp() function represents an exponential function with the natural constant e as the base, and calculates the distance |ji|; Z m ∈R n×n represents sequence association, Softmax(·) represents the attention along the N layer of the current layer model Normalization of the number; the learning scale of the abnormal attention layer is σ∈R n×H , Represents the query, key, and value of the hth head, respectively, and concatenates the outputs of multiple heads And get the final result The anomaly difference uses the symmetric KL difference of prior association and serial association to express the information gain between two distributions, which is expressed as: Among them, KL(·||·) is the m and Z m The KL divergence is calculated between two discrete distributions for each row of The expected value of the logarithmic difference between the original distribution of the data and the probability of the simulated distribution; Dis(P, Z; X)∈R n×1 is the pointwise correlation difference of X with respect to the prior correlation P and sequence correlation Z from multiple layers.

5. The method for detecting anomaly in IoT time series data based on dynamic graph attention according to any one of claims 1 to 4, characterized in that: The dynamic graph neural network module includes a graph structure learning layer and multiple neural network blocks connected in sequence from input to output. The graph structure learning layer is used to capture the spatiotemporal features of the time series and adaptively generate an adjacency matrix. Each neural network block includes a graph neural network GNN and a pooling layer from input to output. The graph neural network GNN is used to model spatiotemporal relationships, and the pooling layer is used to perform soft clustering on the graph neural network GNN.

6. The method for detecting anomaly in IoT time series data based on dynamic graph attention according to claim 5 is characterized in that: The dynamic graph neural network module dynamically models and predicts the time feature matrix to obtain nonlinear prediction results: the adjacency matrix A learned by the graph structure learning layer (i) It is expressed as: A (i) =Embed2(x t ); Among them, Embed2 represents the graph structure learning function; Calculate the similarity matrix between the sampled time series, expressed as: Where distance represents the distance degree; then the dynamic adjacency matrix A( i ) is expressed as: A (i) =f(C (i) W1); Where W1 is the learnable model parameter and f is the activation function; Set the threshold b to make the adjacency matrix sparse, expressed as: For A( i ) Apply normalization; The graph neural network GNN fuses spatial dependencies and temporal patterns to embed the features of nodes and transforms the feature dimensions of nodes into spatial dependencies and temporal patterns, expressed as: where i = 1, 2, ..., n, A (i) encode ∈R n×n Represents the learned node embedding from spatiotemporal modeling. G1(), G2(), and G3() are composed of graph neural network layers GNN and batch normalization layers; The pooling layer uses the diffusion pool Diffpool to train the graph neural network layer GNN and learn the soft clustering allocation of nodes in each layer of the deep graph neural network layer GNN. The process of the pooling layer is expressed as: in, is a trainable parameter matrix representing a linear transformation, and is the assignment matrix representing the projection from the original node to the pooled node, (S (i) ) T Denotes the allocation matrix S (i) The inverse matrix of the distribution matrix S (i) To generate centroids based on the input graph Then calculate the relationship between the centroid of each batch and the encoding graph; Use cosine similarity to evaluate input node embeddings and centroid The relationship between It is expressed as: Among them, cosine similarity () is also called cosine similarity, which is calculated by and The cosine of the angle between them is used to measure their similarity; Row normalization is applied in the resulting distribution matrix, expressed as: Among them, normalize() is row normalization; Cascade Matrix get Is to put multiple Concatenate along a specified axis and perform a trainable weighted sum Γ on the concatenated matrix φ , and get the final allocation matrix S (i) ; After stacking several diffusion pools, the original graph is pooled into a single node and the graph-level representation vector x is obtained. final , which is the result of nonlinear branch training, expressed as: Among them, P1(), P2(), and P3() represent three diffusion pooling layers, and the encoded graph is hierarchically pooled into a single node for generating graph-level representation.

7. The method for detecting anomaly in IoT time series data based on dynamic graph attention according to claim 1 is characterized in that: The linear branch is an autoregressive model.

8. The method for detecting anomaly in IoT time series data based on dynamic graph attention according to claim 1 is characterized in that: In S5, the result of the optimized anomaly detection model analyzing the nonlinear branch of the IoT time series data to be predicted is: The result of linear branching is The results of the nonlinear branch and the linear branch are weighted to obtain the final result. It is expressed as: where Θ is element-wise multiplication; According to the final result Compared with the pre-set abnormal threshold, the final result If it is greater than the abnormal threshold, the output detection result is an abnormal value, otherwise it is a normal value.

9. An IoT time series data anomaly detection system based on dynamic graph attention, characterized in that: include: Storage; The memory is used to store programs; Actuator; The executor is used to execute the program stored in the storage. When the executor executes the program stored in the storage, the method for detecting anomalies in IoT time series data based on dynamic graph attention as described in any one of claims 1 to 8 is implemented.

Images