Multivariable time sequence anomaly detection method and device

Abstract

The invention belongs to the technical field of anomaly detection, and discloses a multivariable time sequence anomaly detection method and device. Preprocessing the abnormal detection data set as input data; establishing an ITtrans detection model classifier, and training the ITtrans detection model classifier according to a loss gradient descent principle; and taking the ITtrans detection model classifier obtained by training the maximum training round as an anomaly detection baseline model, calculating an anomaly score of the to-be-detected sample, and obtaining an anomaly detection result in combination with a preset threshold. An inter-variable and time attention layer module is designed to extract inter-variable correlation and time correlation. Relative relation invariant representation and fusion invariant representation are learned from the angles of variables and time, and the representation gap between a normal point and an abnormal point is enlarged. According to the method, on a reference data set of three multivariable time sequences, ITtrans is compared with the most advanced baseline, and F1 values all reach the best level.

Description

Technical Field

[0001] The present invention relates to the technical field of anomaly detection, and in particular to a multivariate time series anomaly detection method. Background Art

[0002] Time series anomaly detection has been proven to be useful in many fields, such as financial card fraud, industrial intrusion, and medical seizure detection. In each field, the set of values ​​of each sensor over time can be considered a time series, and the combination of these sensor time series is called a multivariate time series. Anomaly detection in multivariate time series is a data mining task that uses multiple indicators to find data points that are significantly different from most other data points.

[0003] The paper "Xu J, Wu H, Wang J, et al. Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy [C] / / International Conference on Learning Representations. 2021." takes advantage of the fact that anomalies cannot establish long-range dependencies in time and proposes an anomaly detection algorithm called Anomaly Transformer. Anomaly Transformer uses the transformer as a framework to model both series association and prior association, and uses the association difference between the two as an anomaly judgment criterion for anomaly detection. However, its method of comparing association differences from a temporal perspective cannot cope with large sections of anomalies within a time window, and the approach of only modeling temporal associations ignores the importance of associations between indicators.

[0004] The paper "Deng A, Hooi B. Graph neural network-based anomaly detection in multivariate time series[C] / / Proceedings of the AAAI conference on artificial intelligence. 2021, 35(5):4027-4035." proposes a graph neural network-based anomaly detection method. This method is based on the phenomenon that abnormal events will disrupt the normal execution logic between sensors. It constructs a graph neural network and predicts the future behavior of the sensor through the attention function of adjacent sensors in the graph to detect and locate anomalies. When faced with complex and diverse real-world scenarios, it is impossible to accurately predict the future behavior of the sensor, and the way of composing the graph according to the time period is easy to ignore fine-grained point anomalies.

[0005] In summary, when faced with complex and diverse abnormal phenomena, existing anomaly detection algorithms tend to consider problems in a one-sided way and ignore the correlation between indicators and time points, resulting in low accuracy, difficulty in application in real scenarios, and low practicality and reference value. Summary of the invention

[0006] The purpose of this invention is to propose a multivariate time series anomaly detection method with high accuracy and applicable to real-world scenarios.

[0007] The technical solution of the present invention is as follows: A multivariate time series anomaly detection method, comprising:

[0008] Preprocess the anomaly detection data set as input data; establish an ITtrans detection model classifier, and train the ITtrans detection model classifier using the input data according to the loss gradient descent principle; use the ITtrans detection model classifier obtained after the maximum training rounds as the anomaly detection baseline model; calculate the anomaly score of the sample to be tested according to the anomaly detection baseline model, and obtain the anomaly detection result in combination with a preset threshold.

[0009] Preprocessing of anomaly detection dataset includes:

[0010] All variable values ​​in the anomaly detection dataset are standardized and normalized according to the following formula;

[0011]

[0012] Among them, r is the original variable value, μ is the mean value of the original variable, s is the standard deviation of the original variable, and z is the standardized variable value; Z min is the minimum value of the original variable, Z max is the maximum value of the original variable, is the normalized variable value;

[0013] The normalized variable values ​​are positionally encoded, and the formula is as follows:

[0014]

[0015] Among them, CNN(·) represents convolution, x represents the variable value after position encoding; PE(·) represents relative position encoding, and the formula is described as follows:

[0016] PE(pos,2i)=sin(pos / 10000 2i / d )

[0017] PE(pos,2i+1)=cos(pos / 10000 2i / d )

[0018] pos represents the position of the moment in the time window, d represents the dimension of the vector embedding, 2i represents the even dimensions in d, and 2i+1 represents the odd dimensions in d.

[0019] The ITtrans detection model classifier includes three network modules: an inter-variable and time transformer encoder, an inter-variable and time transformer decoder, and an output head; input data and a priori association matrix designed by the present invention are input into the inter-variable and time transformer encoder, and the output of the inter-variable and time transformer encoder and the sequence association matrix designed by the present invention are input into the inter-variable and time transformer decoder; the output of the inter-variable and time transformer decoder is input into the output head to obtain the final classification result;

[0020] The inter-variable and temporal transformer encoder and decoder calculate the temporal association and the inter-variable association; a loss function is constructed according to the temporal association and the inter-variable association and the output of the output head; the loss function includes an association loss and a reconstruction loss;

[0021] From the perspective of temporal association, the association loss calculates the symmetric KL divergence of temporal associations between variables and in the temporal transformer encoder and between variables and in the temporal transformer decoder, and finds the association mode of normal points that fuses global information on the basis of maintaining their own characteristics, thereby obtaining temporal association loss; from the perspective of inter-variable association, the association loss is in the form of increasing the association value between strongly associated variables and reducing the association value between unassociated variables, thereby obtaining inter-variable loss;

[0022] The reconstruction loss is in the form of calculating the mean square error between the output of the output head and the input data.

[0023] The inter-variable and temporal transformer encoder includes an embedding layer, an inter-variable and temporal attention layer, a feed-forward layer, and a reversal layer; the structure of the inter-variable and temporal transformer decoder is consistent with the structure of the inter-variable and temporal transformer encoder;

[0024] The input data is fed into the embedding layer for embedding operation, transforming the original tensor of shape W×C into a tensor of shape W×C×D, where W is the size of the time window, C is the number of sensors or machines, and D is the embedding dimension;

[0025] Inter-variable and temporal attention layers and feed-forward layers are stacked alternately; the inter-variable and temporal transformer encoder consists of L layers of inter-variable and temporal attention layers, and the output of the embedding layer As input; the overall equation form of the lth layer is:

[0026]

[0027] in, represents the hidden output of the lth layer of the inter-variable and temporal attention layer; g(·) represents the feed-forward layer; ψ represents the normalization layer; f(·) represents the inter-variable and temporal attention layer module, which is used to calculate the correlation pattern.

[0028] The output of the feedforward layer is inverted to adjust the feature dimension and adjust the W×C×D shape tensor back to W×C.

[0029] The inter-variable and temporal attention layers have a dual-branch structure, namely an inter-variable attention branch and a temporal attention branch; After operation, it is converted into In the inter-variable attention branch, δ = I, and in the temporal branch, δ = T, where I represents the inter-variable attention branch and T represents the temporal attention branch; the input of the l-th temporal attention branch is The calculation process is:

[0030]

[0031]

[0032] in and They represent the prior correlation matrix and the sequence correlation matrix respectively, i and j represent the index of the time window, and the calculation methods are to construct Gaussian kernel and uniform kernel respectively; yes Representations of inter-variable and temporal transformer encoders; yes Representations of inter-variable and temporal transformer decoders;

[0033] Input to the attention branch and The relationship is

[0034] The structure of the l-th layer temporal attention branch and attention branch is expressed as:

[0035]

[0036] in, The query, key, and value for the attention module in the temporal attention branch; are the query, key, and value of the attention module in the inter-variable attention branch, and H is the number of heads; and They are and Parameters; σ is a softmax activation function; It is time-related. is the association between variables; is the output of the temporal attention branch, is the output of the inter-variable attention branch; the output of the inter-variable attention branch and the output of the temporal attention branch are reshaped and fused to obtain the output of the entire inter-variable and temporal attention layer;

[0037] The loss function is as follows:

[0038]

[0039] In the formula express The reconstruction loss of ||·|| F and ||·|| k represents the Frobenius norm and k-norm; λ is the loss balance term; IntDis represents the inter-variable correlation loss, and TemDis represents the temporal correlation loss;

[0040] The strength of the association between variables is divided into strong association, weak association and no association; the similarity between the time series of each variable is calculated to obtain the association matrix ρ;

[0041]

[0042] in, Represents the set of the pth indicator in the current time window; the window size is τ; express and The covariance of Var(f p ) is f p The variance of , ε is a positive number; M represents the total number of time windows.

[0043] Use k-means clustering on ρ to obtain the locations of strongly associated variables, weakly associated variables, and unassociated variables; operate the association between variables based on the clustering results; the association loss between variables is calculated as:

[0044]

[0045] in, and They represent the sum of the values ​​of the uncorrelated positions at the lth layer and the sum of the values ​​of the strongly correlated positions at the lth layer respectively;

[0046] Define the temporal associations between variables and the lth layer in the temporal transformer encoder and temporal associations between variables and the temporal transformer decoder layer l The similarity measure is where KL(·||·) represents the KL divergence; the temporal association loss between the inter-variable and temporal transformer encoder and the inter-variable and temporal transformer decoder is calculated as:

[0047]

[0048] The steps for obtaining the anomaly detection result are as follows:

[0049] Compute the inter-variable attention branch scores:

[0050]

[0051] Calculate the total score:

[0052]

[0053] Where W is element-by-element multiplication; express Element-by-element exception criteria;

[0054] The anomaly score of each sequence sample to be tested after the anomaly detection baseline model is compared with the set detection threshold to obtain the final sample category CI, where the sample is classified as a positive class when the anomaly score is less than the threshold, otherwise it is classified as a negative class;

[0055]

[0056] Detect the samples of the sequence to be tested one by one to obtain the abnormal detection results of the attack behavior in the data;

[0057] A multivariate time series anomaly detection device, comprising:

[0058] The data processing module pre-processes the anomaly detection data set as input data;

[0059] Storage module, used to store ITtrans detection model classifiers and anomaly detection baseline models;

[0060] The execution module is used to train the ITtrans detection model classifier; train the ITtrans detection model classifier according to the loss gradient descent principle based on the input data; the ITtrans detection model classifier obtained after the maximum training rounds is used as the anomaly detection baseline model and stored in the storage module; calculate the anomaly score of the sample to be tested according to the anomaly detection baseline model, and obtain the anomaly detection result in combination with a preset threshold.

[0061] The ITtrans detection model classifier includes three network modules: an inter-variable and temporal transformer encoder, an inter-variable and temporal transformer decoder, and an output head; input data and a priori association matrix are input into the inter-variable and temporal transformer encoder, and the output of the inter-variable and temporal transformer encoder and the sequence association matrix are input into the inter-variable and temporal transformer decoder; the output of the inter-variable and temporal transformer decoder is input into the output head to obtain the final classification result;

[0062] The inter-variable and temporal transformer encoder and decoder calculate the temporal association and the inter-variable association; a loss function is constructed according to the temporal association and the inter-variable association and the output of the output head; the loss function includes an association loss and a reconstruction loss;

[0063] From the perspective of temporal association, the association loss calculates the symmetric KL divergence of temporal associations between variables and in the temporal transformer encoder and between variables and in the temporal transformer decoder, and finds the association mode of normal points that fuses global information on the basis of maintaining their own characteristics, thereby obtaining temporal association loss; from the perspective of inter-variable association, the association loss is in the form of increasing the association value between strongly associated variables and reducing the association value between unassociated variables, thereby obtaining inter-variable loss;

[0064] The reconstruction loss is in the form of calculating the mean square error between the output of the output head and the input data.

[0065] Beneficial effects of the present invention:

[0066] (1) An inter-variable and temporal attention layer module is designed to extract inter-variable and temporal associations. Relative relation invariant representation and fusion invariant representation are learned from inter-variable and temporal perspectives, respectively, which widens the representation gap between normal points and abnormal points.

[0067] (2) The loss function consists of three parts: From a temporal perspective, it compares the correlation similarity between the encoder and decoder. From a variable perspective, it strengthens the correlation between strongly correlated variables and weakens the correlation between uncorrelated variables. The reconstruction loss calculates the mean square error between the output of the output head and the input data.

[0068] The present invention considers that when an anomaly occurs, it will simultaneously destroy the normal variables and time associations in the system, and finds that: on the one hand, the normal operation of the system is periodic, and the normal point can integrate the information of the entire sequence; on the other hand, from the perspective of the entire sequence, when the system is operating normally, the relationship between variables remains fixed. The present invention models these two characteristics and expands the gap between normal points and abnormal points. Experiments show that the model ITtrans proposed in the present invention achieves the best F1 value when compared with the most advanced baseline models on three benchmark datasets of multivariate time series. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a flow chart of a multivariate time series anomaly detection method;

[0070] Figure 2 This is the framework diagram of the ITtrans detection model classifier;

[0071] Figure 3 It is the framework diagram of inter-variable and temporal transformer encoder;

[0072] Figure 4(a) is a schematic diagram of time-correlated loss;

[0073] Figure 4(b) is a schematic diagram of the correlation loss between variables.

[0074] Figure 5 Schematic diagram of the multivariate time series anomaly detection setup. DETAILED DESCRIPTION

[0075] The specific implementation of the present invention is further described in detail below in conjunction with the drawings and examples.

[0076] When anomalies occur in time series data, there are the following characteristics: on the one hand, due to the periodicity of the time series, there is a significant correlation between each time point and other time points, and normal points can fuse the time information of the entire sequence while maintaining their own characteristics. On the contrary, anomalies are similar to "sudden insertions" and have no correlation with other normal points. On the other hand, from the perspective of the entire system, the correlation pattern between variables is fixed. When an anomaly occurs, the correlation is broken, which weakens the correlation between variables with strong correlations, or strengthens the correlation between variables without correlations. Based on the above characteristics, the present invention proposes an ITtrans detection model classifier.

[0077] The ITtrans detection model classifier proposed in the present invention simultaneously models the correlation between variables and the temporal correlation, is more robust, and can adapt to various scenarios. The anomaly evaluation index proposed in the present invention combines the anomaly score between variables, the temporal anomaly score, and the reconstruction score, which makes the evaluation of anomalies more comprehensive. The present invention fully considers the actual application situation and does not require manual intervention in multivariate time series data. The proposed method is an automated detection method.

[0078] This embodiment uses three public multi-sensor data sets in different scenarios (aerospace and water treatment system scenarios) to verify the model. The specific data are shown in Table 1.

[0079] Table 1

[0080]

[0081] A multivariate time series anomaly detection method, such as Figure 1 As shown, the steps are as follows:

[0082] Step 1: Preprocess the anomaly detection data set to obtain the input data of the ITtrans detection model classifier proposed in the present invention. The specific steps are as follows:

[0083] Step 1.1: Standardize all variable values. The processing formula is as follows:

[0084]

[0085] Where r is the original variable value, μ is the mean value of the variable, s is the standard deviation of the variable, and z is the standardized variable value.

[0086] Step 1.2 maps the standardized values ​​to the [0,1] interval through normalization. The normalization formula is as follows:

[0087]

[0088] Where Z min is the minimum value of the variable, Z max is the maximum value of the variable, is the normalized variable value.

[0089] Step 1.3 performs position encoding on the normalized variable values. The formula is as follows:

[0090]

[0091] Among them, CNN(·) represents convolution, x represents the variable value after position encoding. PE(·) represents relative position encoding, and the formula is described as follows:

[0092] PE(pos,2i)=sin(pos / 10000 2i / d )

[0093] PE(pos,2i+1)=cos(pos / 10000 2i / d )

[0094] pos represents the position of the moment in the time window, d represents the dimension of the vector embedding, 2i represents the even dimensions in d, and 2i+1 represents the odd dimensions in d.

[0095] Step 2: Use the converted data as input data to train the ITtrans detection model classifier. The overall framework of the model is as follows: Figure 2 As shown, it mainly includes inter-variable and temporal transformer encoder, inter-variable and temporal transformer decoder and output head. The specific idea of ​​this step is:

[0096] Step 2.1 The preprocessed variable values ​​are input into the inter-variable and temporal transformer encoder. The inter-variable and temporal transformer encoder is as follows Figure 3 As shown, the detailed process is:

[0097] The embedding operation has the effect of transforming the original tensor of shape W×C into a tensor of shape W×C×D, where W is the size of the time window, C is the number of sensors or machines, and D is the embedding dimension.

[0098] In the time branch, the lth layer The calculation process is:

[0099]

[0100] in and They represent the prior correlation matrix and the sequence correlation matrix respectively, i and j represent the index of the time window, and the calculation methods are to construct Gaussian kernel and uniform kernel respectively; yes Representations of inter-variable and temporal transformer encoders; yes Representations of inter-variable and temporal transformer decoders;

[0101] Inter-variable and temporal attention and feed-forward layers are stacked alternately. Assuming the encoder or decoder contains L layers, the input time series The overall equation form of the lth layer is:

[0102]

[0103] in represents the hidden output of the inter-variable branch or the l-th layer of the temporal branch. g(·) represents the feedforward. ψ represents the normalization layer. f(·) represents the inter-variable and temporal attention module, which is used to calculate the correlation pattern.

[0104] The inter-variable and temporal attention layers have a dual-branch structure. After linear and reshaping operations, it is converted into The inter-variable attention branch and the temporal attention branch are input respectively, where δ = I in the inter-variable attention branch and δ = T in the temporal attention branch. Then, the outputs of the two branches are reshaped and fused to obtain the output of the entire inter-variable and temporal attention layer. The structure of each branch in the lth layer is expressed by the formula:

[0105]

[0106] in and are the query, key, and value of the attention module in the time branch and inter-variable branch, respectively, and H is the number of heads. and They are and Parameters of σ. σ is a softmax activation function. and They are temporal correlation and inter-variable correlation respectively. are the outputs of the two branches respectively. In the inter-variable branch, and The relationship is

[0107] Perform an inversion operation, adjust the feature dimension, find the embedded low-dimensional representation of each point, and adjust the W×C×D shape tensor back to W×C.

[0108] Step 2.2 inputs the output of the inter-variable and temporal transformer encoder into the inter-variable and temporal transformer decoder, and the structure of the inter-variable and temporal transformer decoder is consistent with that of the inter-variable and temporal transformer encoder.

[0109] Step 2.3 inputs the output of the inter-variable and temporal transformer decoders into the output head to obtain the final result. The output head is a linear layer used to adjust the dimension of the output.

[0110] Step 2.4 calculates the loss. From the perspective of time, the loss is constructed based on KL divergence, and the association pattern of normal points that can integrate global information while maintaining their own characteristics is found. From the perspective of indicators, the loss is in the form of increasing the association value between strongly correlated variables and reducing the association value between uncorrelated variables.

[0111] The calculation process of the difference in association patterns between variables is as follows:

[0112] The correlation strength between variables can be simplified into three types: strong correlation, weak correlation, and no correlation. First, calculate the similarity between the time series of each variable to obtain the correlation matrix ρ. The calculation process is as follows:

[0113]

[0114] in, Represents the set of the pth indicator in the current time window; the window size is τ; express and The covariance of Var(f p ) is f p The variance of , ε is a positive number; M represents the total number of time windows.

[0115] Then, k-means clustering is used to obtain the locations of strong, weak, and uncorrelated variables. The inter-variable attention module is associated with the clustering results. The inter-variable association loss is shown in Figure 4(b):

[0116]

[0117] in, and They represent the sum of the values ​​of the unassociated positions and the strongly associated positions in the lth layer respectively.

[0118] The time loss calculation process is as follows:

[0119] Define the l-th layer association in the encoder Associated with decoder layer l The similarity measure is Where KL(·||·) represents the KL divergence. The temporal correlation loss of ITtrans is shown in Figure 4(a), and the calculation process is:

[0120]

[0121] The total loss is defined as follows:

[0122]

[0123] In the formula express Reconstruction of. ||·|| F and ||·|| k represents the Frobenius norm and k-norm. λ is the loss balance term.

[0124] Step 2.5: Repeat steps 2.1-2.4 until the model reaches the maximum number of training rounds.

[0125] Step 3: The anomaly detection baseline model obtained after normal sample training calculates the anomaly score of the sample to be tested, and completes anomaly detection in combination with the preset threshold. The detailed steps are as follows:

[0126] Step 3.1 Calculate the inter-variable branching score:

[0127]

[0128] Step 3.2 Calculate the total score:

[0129]

[0130] where W is the element-wise multiplication. express Element-wise exception criteria.

[0131] Step 3.3 compares the anomaly score of each sequence sample to be tested after the training model with the detection threshold to obtain the final sample category CI, where the anomaly score is less than the threshold and is classified as a positive class, otherwise it is classified as a negative class.

[0132]

[0133] Step 3.4: Repeat steps 3.1-3.3 for each sample to be tested to achieve anomaly detection of attack behaviors in the data.

[0134] like Figure 5 As shown, a multivariate time series anomaly detection device comprises:

[0135] The data processing module pre-processes the anomaly detection data set as input data;

[0136] A storage module, used to store ITtrans detection model classifiers;

[0137] The execution module is used to train the ITtrans detection model classifier; train the ITtrans detection model classifier according to the loss gradient descent principle; the ITtrans detection model classifier obtained after the maximum training rounds is used as the anomaly detection baseline model and stored in the storage module; calculate the anomaly score of the sample to be tested, and combine it with the preset threshold to obtain the anomaly detection result.

[0138] In order to verify the effectiveness of the method proposed in the present invention, a set of experiments were conducted to compare the detection effects of the system model with two more advanced classification algorithms, including AnomalyTransformer based on time correlation bias and GDN based on graph neural network, on data sets of various scenarios. The model of the present invention is superior to other compared models in all evaluation indicators.

[0139] (1) In terms of the detection performance of the model, since the present invention simultaneously considers the natural differences in the inter-variable and temporal correlation patterns between normal points and abnormal points, and designs inter-variable and temporal losses to amplify the differences between normal points and abnormal points, the model detection effect is ultimately better. As can be seen from Table 2, whether it is any one of the three accuracy, recall rate, and F1 value, the detection model proposed by the present invention is relatively superior to other methods, thus proving that the present invention has a great advantage in detection performance.

[0140] Table 2

[0141]

[0142] (2) In terms of the universality of the detection model, the present invention uses two scenarios and three representative multi-sensor data sets with different numbers of sensors to verify the method. The existing detection models either only focus on the relationship between variables in multivariate time series, such as the GDN model; or only focus on the time relationship, such as the AnomalyTransformer, resulting in poor generalization performance of the model in different scenarios. This model comprehensively considers the characteristics of multivariate time series in two aspects and is suitable for anomaly detection tasks in different scenarios.

[0143] In view of the above observations, the system model proposed in the present invention takes the normalized and position-encoded multivariate time series as input, passes through the encoder and decoder in sequence, obtains the inter-variable correlation pattern, the time correlation pattern and the reconstructed output, and when performing anomaly score judgment, combines the inter-variable correlation pattern score, the time correlation pattern score and the reconstructed correlation pattern score to achieve the correct classification of anomalies in different industrial scenario data.

Claims

1. A multivariate time series anomaly detection method, characterized in that: include: Preprocess the anomaly detection dataset as input data; Establishing an ITtrans detection model classifier, and training the ITtrans detection model classifier according to the loss gradient descent principle using the input data; The ITtrans detection model classifier obtained after the maximum number of training rounds is used as the anomaly detection baseline model; the anomaly score of the sample to be tested is calculated according to the anomaly detection baseline model, and the anomaly detection result is obtained by combining the preset threshold.

2. The multivariate time series anomaly detection method according to claim 1, characterized in that: The ITtrans detection model classifier includes three network modules: an inter-variable and temporal transformer encoder, an inter-variable and temporal transformer decoder, and an output head; input data and a priori association matrix are input into the inter-variable and temporal transformer encoder, and the output of the inter-variable and temporal transformer encoder and the sequence association matrix are input into the inter-variable and temporal transformer decoder; the output of the inter-variable and temporal transformer decoder is input into the output head to obtain the final classification result; The inter-variable and temporal transformer encoder and decoder calculate the temporal association and the inter-variable association; a loss function is constructed according to the temporal association and the inter-variable association and the output of the output head; the loss function includes an association loss and a reconstruction loss; From the perspective of temporal association, the association loss calculates the symmetric KL divergence of temporal associations between variables and in the temporal transformer encoder and between variables and in the temporal transformer decoder, and finds the association mode of normal points that fuses global information on the basis of maintaining their own characteristics, thereby obtaining temporal association loss; from the perspective of inter-variable association, the association loss is in the form of increasing the association value between strongly associated variables and reducing the association value between unassociated variables, thereby obtaining inter-variable loss; The reconstruction loss is in the form of calculating the mean square error between the output of the output head and the input data.

3. The multivariate time series anomaly detection method according to claim 1 or 2, characterized in that: The inter-variable and temporal transformer encoder includes an embedding layer, an inter-variable and temporal attention layer, a feed-forward layer, and a reversal layer; the structure of the inter-variable and temporal transformer decoder is consistent with the structure of the inter-variable and temporal transformer encoder; The input data is fed into the embedding layer for embedding operation, transforming the original tensor of shape W×C into a tensor of shape W×C×D, where W is the size of the time window, C is the number of sensors or machines, and D is the embedding dimension; Inter-variable and temporal attention layers and feed-forward layers are stacked alternately; the inter-variable and temporal transformer encoder consists of L layers of inter-variable and temporal attention layers, and the output of the embedding layer As input; the overall equation form of the lth layer is: in, represents the hidden output of the lth layer of the inter-variable and temporal attention layer; g(·) represents the feed-forward layer; ψ represents the normalization layer; f(·) represents the inter-variable and temporal attention layer module, which is used to calculate the correlation pattern; The output of the feedforward layer is inverted to adjust the feature dimension and adjust the W×C×D shape tensor back to W×C.

4. The multivariate time series anomaly detection method according to claim 3, characterized in that: The inter-variable and temporal attention layers have a dual-branch structure, namely an inter-variable attention branch and a temporal attention branch; After operation, it is converted into In the inter-variable attention branch, δ = I, and in the time branch, δ = T, where I represents the inter-variable attention branch and T represents the time attention branch; Input of the l-th layer temporal attention branch The calculation process is: in and They represent the prior correlation matrix and the sequence correlation matrix respectively, i and j represent the index of the time window, and the calculation methods are to construct Gaussian kernel and uniform kernel respectively; yes Representations of inter-variable and temporal transformer encoders; yes Representation of the transformer decoder across variables and time; input to the attention branch and The relationship is The structure of the l-th layer temporal attention branch and attention branch is expressed as: in, The query, key, and value for the attention module in the temporal attention branch; are the query, key, and value of the attention module in the inter-variable attention branch, and H is the number of heads; and They are and Parameters; σ is a softmax activation function; It is time-related. is the association between variables; is the output of the temporal attention branch, is the output of the inter-variable attention branch; the output of the inter-variable attention branch and the output of the temporal attention branch are reshaped and fused to obtain the output of the entire inter-variable and temporal attention layer.

5. The multivariate time series anomaly detection method according to claim 2, characterized in that: The loss function is as follows: In the formula express The reconstruction loss of ||·|| F and ||·|| k represents the Frobenius norm and k-norm; λ is the loss balance term; IntDis represents the inter-variable correlation loss, and TemDis represents the temporal correlation loss.

6. The multivariate time series anomaly detection method according to claim 5, characterized in that: The strength of the association between the variables is divided into strong association, weak association and no association; the similarity between the time series of each variable is calculated to obtain the association matrix ρ; in, Represents the set of the pth indicator in the current time window; the window size is τ; express and The covariance of Var(f p ) is f p The variance of , ε is a positive number; M represents the total number of time windows, Use k-means clustering on ρ to obtain the locations of strongly associated variables, weakly associated variables, and unassociated variables; operate the association between variables based on the clustering results; the association loss between variables is calculated as: in, and They represent the sum of the values ​​of the uncorrelated positions in the lth layer and the sum of the values ​​of the strongly correlated positions in the lth layer respectively.

7. The multivariate time series anomaly detection method according to claim 5, characterized in that: The time-correlated loss calculation process is as follows: Define the temporal associations between variables and the lth layer in the temporal transformer encoder and temporal associations between variables and the temporal transformer decoder layer l The similarity measure is where KL(·||·) represents the KL divergence; the temporal association loss between the inter-variable and temporal transformer encoder and the inter-variable and temporal transformer decoder is calculated as:

8. The multivariate time series anomaly detection method according to claim 6 or 7, characterized in that: The steps for obtaining the anomaly detection result are as follows: Compute the inter-variable attention branch scores: Calculate the total score: Among them, ⊙ is element-by-element multiplication; express Element-by-element exception criteria; The anomaly score of each sequence sample to be tested after the anomaly detection baseline model is compared with the set detection threshold to obtain the final sample category CI, where the sample is classified as a positive class when the anomaly score is less than the threshold, otherwise it is classified as a negative class; The samples of the sequence to be tested are tested one by one to obtain the abnormal detection results of the attack behavior in the data.

9. A multivariate time series anomaly detection device, characterized in that: include: The data processing module pre-processes the anomaly detection data set as input data; A storage module, used to store ITtrans detection model classifiers; The execution module is used to train the ITtrans detection model classifier; train the ITtrans detection model classifier according to the loss gradient descent principle based on the input data; the ITtrans detection model classifier obtained after the maximum training rounds is used as the anomaly detection baseline model and stored in the storage module; calculate the anomaly score of the sample to be tested according to the anomaly detection baseline model, and obtain the anomaly detection result in combination with a preset threshold.

10. The multivariate time series anomaly detection device according to claim 9, characterized in that: The ITtrans detection model classifier includes three network modules: an inter-variable and temporal transformer encoder, an inter-variable and temporal transformer decoder, and an output head; input data and a priori association matrix are input into the inter-variable and temporal transformer encoder, and the output of the inter-variable and temporal transformer encoder and the sequence association matrix are input into the inter-variable and temporal transformer decoder; the output of the inter-variable and temporal transformer decoder is input into the output head to obtain the final classification result; The inter-variable and temporal transformer encoder and decoder calculate the temporal association and the inter-variable association; a loss function is constructed according to the temporal association and the inter-variable association and the output of the output head; the loss function includes an association loss and a reconstruction loss; From the perspective of temporal association, the association loss calculates the symmetric KL divergence of temporal associations between variables and in the temporal transformer encoder and between variables and in the temporal transformer decoder, and finds the association mode of normal points that fuses global information on the basis of maintaining their own characteristics, thereby obtaining temporal association loss; from the perspective of inter-variable association, the association loss is in the form of increasing the association value between strongly associated variables and reducing the association value between unassociated variables, thereby obtaining inter-variable loss; The reconstruction loss is in the form of calculating the mean square error between the output of the output head and the input data.

Images