An Abnormal Detection Method for Power Dispatching Monitoring Data Based on Spatiotemporal Anomaly Transformer

Abstract

An embodiment of the present invention proposes an abnormal detection method for power dispatching monitoring data based on spatio-temporal anomaly Transformer, including: constructing a spatio-temporal anomaly Transformer model, taking historical power dispatching monitoring data as the input of the model, calculating the reconstruction error, the time-dimensional correlation deviation, and the space-dimensional correlation deviation; combining the reconstruction error, the time-dimensional correlation deviation, and the space-dimensional correlation deviation to calculate the loss of the model, and using the loss as the optimization objective to train the model; taking the power dispatching monitoring data to be detected as the input of the model, calculating the reconstruction error, the time-dimensional correlation deviation, and the space-dimensional correlation deviation; combining the reconstruction error, the time-dimensional correlation deviation, and the space-dimensional correlation deviation to calculate the anomaly score of the input data, and accordingly determining whether the input power dispatching monitoring data is abnormal. The technical solution provided by the embodiment of the present invention can improve the accuracy of abnormal detection of power dispatching monitoring data.

Description

Technical Field

[0001] The present invention relates to a method for detecting anomalies in power dispatching monitoring data, and particularly to a method for detecting anomalies in power dispatching monitoring data based on spatio-temporal anomaly Transformer.

Background Art

[0002] With the rapid development of the global economy, the intelligent global energy Internet has become the direction and trend of the development of the power industry. The goal of the smart grid is to build a panoramic real-time system covering multiple links in the power system, including power generation, transmission, transformation, distribution, power consumption, and dispatching. By using advanced technologies such as information communication, computer, and control, while improving the operating efficiency of each part of the power system, reducing costs and environmental pollution, the reliability, stability, and self-healing ability of the power system are improved as much as possible. As the "brain" of the power grid, the power dispatching automation system integrates power grid data collection, storage, analysis, decision-making, and control, and is the center to ensure the safe and stable operation of the power grid.

[0003] Although the power dispatching automation system can collect and analyze a large amount of operation data, enabling operation and maintenance personnel to efficiently and accurately evaluate the state of the power grid, as the "brain" of the power grid, once the business anomalies in the power dispatching automation system are not discovered and properly handled in a timely manner, it will seriously threaten the stable operation of the power grid and may even lead to the paralysis of the power grid. The business of the power dispatching automation system consists of different processes, and the system will monitor multiple performance indicators of relevant hardware and processes in real time. By detecting whether these monitoring data are abnormal, the operating state of each business in the system can be reflected. Therefore, in order to ensure the safe and reliable operation of the smart grid, the anomaly detection technology for power dispatching monitoring data becomes increasingly important.

[0004] Since the system collects performance indicator data in real time according to a fixed frequency, the data of adjacent sampling points have strong temporal continuity, so the power dispatch monitoring data can be regarded as a time series. In addition, due to the characteristics of the power dispatch automation system, such as the close coupling of business and hardware, the numerous business, and the complex business hierarchical relationship, the attribute feature dimension of its monitoring data is generally high and multiple features are interrelated. The method of setting a static threshold for each monitoring indicator fails to consider the complex correlation between the monitoring quantities, and is prone to false alarms or omissions of abnormal conditions; the traditional anomaly detection method assumes that the data are independent of each other and cannot directly consider the temporal continuity in the monitoring data. Therefore, it is necessary to introduce a multi-dimensional time series anomaly detection method based on deep learning to detect anomalies in monitoring data more accurately and timely. With the widespread application of technologies such as sensors and the Internet of Things in power grid systems, the dispatch automation system can collect and store massive amounts of historical monitoring data. Since the dispatch automation system has high robustness, it takes a lot of cost and time to collect and manually annotate a sufficient amount of abnormal data. Therefore, the unsupervised multi-dimensional time series anomaly detection method that only needs to use historical monitoring data under normal operating conditions for training is more suitable for power dispatch automation systems. The attribute feature dimension of power dispatch monitoring data is generally high and multiple features are interrelated. When an anomaly occurs, the temporal correlation of the data and the correlation between features will change. The existing multidimensional time series anomaly detection method based on deep learning can only extract the semantic information of the time dimension and space dimension in a single order, and cannot well capture the change of the correlation between features when an anomaly occurs. It does not fully utilize the correlation between features for anomaly detection, resulting in limited detection ability of business anomalies. Therefore, an unsupervised multidimensional time series anomaly detection algorithm is proposed, which can more fully extract the semantic information in the monitoring data and more effectively utilize the temporal correlation and the correlation between features in the monitoring data to improve the accuracy of anomaly detection, which is of great significance for strengthening the state monitoring of the power grid and ensuring the safety of the power grid. [Summary of the invention]

[0005] In view of this, the present invention proposes an anomaly detection method for power dispatching monitoring data based on spatiotemporal anomaly Transformer to improve the performance of anomaly detection in power dispatching monitoring data.

[0006] The present invention proposes a method for detecting anomaly in power dispatching monitoring data based on spatiotemporal anomaly Transformer, which comprises the following steps:

[0007] (1) A spatiotemporal anomaly Transformer model is constructed. The historical data of power dispatch monitoring is used as the input of the model. The reconstruction error, time dimension correlation deviation, and space dimension correlation deviation of the input data are calculated. Specifically,

[0008] Use the historical data of the operating conditions of the hardware and processes related to the power dispatching system collected by the power dispatching monitoring system as the training set of the model, denoted as Among them represents the set of real numbers, N represents the features of the data, specifically including the CPU usage rate, memory usage rate, disk I / O, network I / O of the server, and the CPU occupancy rate, memory occupancy rate, disk I / O, network I / O of each related process. T train is the total number of data in the historical data set χ train ;

[0009] Use a sliding window with length L and step size O to divide the historical data set χ train into multiple time windows, and the values of L and O are both 100. Denote the last data as x t The time window is X t ={x t-L+1 ,x t-L+2 ,...,x t}. Build a spatio-temporal anomaly Transformer model and calculate the reconstruction error, time-dimensional correlation deviation and space-dimensional correlation deviation of the input data:

[0010] The spatio-temporal anomaly Transformer model consists of two parts: a space-time branch and a time-space branch. The space-time branch is composed of a spatial Transformer and a time anomaly Transformer connected end to end, and the time-space branch is composed of a time Transformer and a space anomaly Transformer connected end to end.

[0011] The spatial Transformer in the space-time branch is composed of M spatial Transformer layers, and the value of M is 3. Each spatial Transformer layer is composed of multi-head spatial self-attention, residual connection, two normalization layers and a feed-forward network. Denote the number of heads of the multi-head spatial self-attention as H, and the value of H is 8. The calculation process of the h-th spatial self-attention head in the m-th spatial Transformer layer is as follows:

[0012]

[0013]

[0014] Z m,h =S m,h V m,h

[0015] Among them, is the output of the previous spatial Transformer layer, and the input of the first spatial Transformer layer is the time window X t is the transpose of and are three different linear layers, Q m,h , K m,h and V m,h are query, key, and value respectively, d ST is the dimension of query, key, and value, d ST takes values as represents floor function. Softmax(·) is the normalized exponential function, S m,h is the attention weight, Z m,h is the output of this spatial self-attention head.

[0016] The output of the m-th spatial Transformer layer is obtained by combining the outputs of all H spatial self-attention heads through residual connection, two normalization layers, and a feed-forward network. The process is as follows:

[0017]

[0018]

[0019]

[0020] where Concat(·) represents concatenation, is a linear layer, Z m is the output of the multi-head spatial self-attention, LayerNorm(·) represents normalization, is the output of the first normalization layer, FFN(·) represents the feed-forward network, is the output of this spatial Transformer layer.

[0021] The temporal anomaly Transformer in the spatio-temporal branch consists of M temporal anomaly Transformer layers. Each temporal anomaly Transformer layer is composed of multi-head temporal anomaly self-attention, residual connection, two normalization layers, and a feed-forward network. The number of heads of the multi-head temporal anomaly self-attention is also H, and the dimensions of the query, key, and value are d TAT , d TAT takes values as The calculation process of the temporal anomaly Transformer is basically the same as that of the spatial Transformer. On this basis, the temporal anomaly Transformer needs to additionally calculate the temporal dimension correlation deviation. The process is as follows:

[0022]

[0023]

[0024]

[0025] Among them, is the output of the previous time anomaly Transformer layer, and the input of the first time anomaly Transformer layer is the transpose of the output of the spatial Transformer. is a linear layer, σ m,h When calculating the prior correlation in the time dimension, is the variance corresponding to the i-th data in the time window. Rescale(·) means scaling the input to a sum of 1 by dividing by the sum, Exp(·) means calculating the exponential with base e, i and j represent the i-th and j-th data in the time window, and P m,h is the prior correlation in the time dimension, and P i m,h is the prior correlation in the time dimension corresponding to the i-th data in the time window. is the attention weight corresponding to the i-th data in the time window. KL(·) means calculating the KL divergence, and TAD(X t ) is the time dimension correlation deviation of the time window X t .

[0026] The time Transformer in the time - space branch consists of M time Transformer layers. Each time Transformer layer consists of multi - head time self - attention, residual connection, two normalization layers, and a feed - forward network. The number of heads of the multi - head time self - attention is also H, and the dimensions of the query, key, and value are d TT , d TT takes the value of The calculation method of the time Transformer is the same as that of the spatial Transformer. The input of the first time Transformer layer is the time window X t .

[0027] The spatial anomaly Transformer in the time - space branch consists of M spatial anomaly Transformer layers. Each spatial anomaly Transformer layer consists of multi - head spatial anomaly self - attention, residual connection, two normalization layers, and a feed - forward network. The number of heads of the multi - head spatial anomaly self - attention is also H, and the dimensions of the query, key, and value are d SAT , d SAT takes the value of The calculation process of the Spatial Anomaly Transformer is basically the same as that of the Spatial Transformer. The input of the first Spatial Anomaly Transformer layer is the transpose of the output of the Temporal Transformer. On this basis, the Spatial Anomaly Transformer needs to additionally calculate the spatial dimension correlation deviation, and the process is as follows:

[0028]

[0029] Among them, C m,h is the prior spatial dimension correlation of the h-th spatial anomaly self-attention head in the m-th Spatial Anomaly Transformer layer, and C m,h is a randomly initialized matrix. is the prior spatial dimension correlation of the i-th feature to the j-th feature, is the attention weight of the i-th feature to the j-th feature, and SAD(X t ) is the spatial dimension correlation deviation of the time window X t .

[0030] The model combines the output of the Temporal Anomaly Transformer and the output of the Spatial Anomaly Transformer to reconstruct the time window X t through a linear layer and calculates the reconstruction error, and the process is as follows:

[0031]

[0032]

[0033]

[0034] Among them, is the output of the Temporal Anomaly Transformer, is the transpose of the output of the Spatial Anomaly Transformer, X CAT is the result after concatenating the two outputs, W RCO is the linear layer used in reconstruction, Sigmoid(·) represents the Sigmoid activation function, is the reconstruction of the time window X t , X t,n is the n-th feature in the time window X t , is the n-th feature in the reconstructed time window , is the reconstruction error.

[0035] (2) Calculate the loss of the model by combining the reconstruction error, the time - dimension correlation deviation, and the space - dimension correlation deviation, and use the loss as the optimization objective to train the model. Specifically:

[0036] Based on the reconstruction error obtained in step (1) The time - dimension correlation deviation TAD(X t ) and the space - dimension correlation deviation SAD(X t ), calculate the loss of the spatio - temporal anomaly Transformer model. The process is as follows:

[0037]

[0038]

[0039] Among them, k1, k2, and k3 are the coefficients of each item, and the values of k1, k2, and k3 are 10, 10, and 0.1 respectively. L max (X t ) is the loss of the model in the maximization stage during training, and L min (X t ) is the loss of the model in the minimization stage during training.

[0040] Use the Adam optimizer to train the network parameters and the space - dimension prior correlation C m,h in the spatio - temporal anomaly Transformer model according to the optimization objective. The number of training iterations is E, the batch size is B, and the learning rate is lr. The value of E is 500, the value of B is 32, and the value of lr is 0.0001. Each iteration during training is divided into two parts: the maximization stage and the minimization stage. In the maximization stage, the optimization objective of the model is:

[0041]

[0042] In the minimization stage, the optimization objective of the model is:

[0043]

[0044] During the training process, if L max (X t ) and L min (X t ) do not decrease within E lr iterations, then the learning rate lr is reduced to half of the original. If L max (X t ) and L min (X t ) do not decrease within E es iterations, then the training process is terminated prematurely. The value of E lr is 3, Ees The value of

[0045] (3) Take the power dispatching monitoring data to be detected as the input of the model, and calculate the reconstruction error, the time - dimension correlation deviation, and the space - dimension correlation deviation. Specifically:

[0046] Denote the power dispatching monitoring data to be detected as x test Each feature in train is the same as the feature in the historical power dispatching monitoring data χ test . Combine x test and the power dispatching monitoring data collected at the previous L - 1 time points to form a time window X test , and input X test into the trained spatio - temporal anomaly Transformer model, and repeat step (1) to calculate the reconstruction error of X test the time - dimension correlation deviation TAD(X test ).

[0047] (4) Combine the reconstruction error, the time - dimension correlation deviation, and the space - dimension correlation deviation to calculate the anomaly score of the input data, and accordingly determine whether the input power dispatching monitoring data is abnormal. Specifically:

[0048] Based on the reconstruction error test the time - dimension correlation deviation TAD(X ) and the space - dimension correlation deviation SAD(X test ) of the time window X test obtained in step (3), calculate the anomaly score of each data in X test . The process is as follows:

[0049]

[0050] where ⊙ represents element - by - element multiplication of matrices, λ is the coefficient of the time - dimension correlation deviation TAD(X test ), and the value of λ is 0.01. Score(X test ) is the anomaly score of each data in the time window X test . The last anomaly score Score(x test ) in Score(X test ) is the anomaly score corresponding to the data x test to be detected.

[0051] Repeat step (4) to calculate the anomaly scores Score(χ train ) of all the historical power dispatching monitoring data χ train), take the maximum value in Score(χ train ) as the threshold thr. If the anomaly score Score(x test ) of the data x to be detected test ) is greater than thr, then the data x to be detected test is abnormal, realizing the anomaly detection of power dispatching monitoring data.

[0052] The anomaly detection method for power dispatching monitoring data improves the accuracy of anomaly detection for power dispatching monitoring data.

[0053] As can be seen from the above technical solutions, the present invention has the following beneficial effects:

[0054] In the technical solution implemented by the present invention, by using the two orders of time-space branch and space-time branch to extract semantic information, the time-dimensional semantic information and space-dimensional semantic information in the monitoring data can be more fully extracted; by using the time anomaly Transformer and space anomaly Transformer to explicitly calculate the correlation deviation between time points and the correlation deviation between features, the time correlation and feature correlation in the monitoring data can be more effectively utilized for anomaly detection, enhancing the method's ability to judge anomalies, thereby improving the accuracy of anomaly detection for power dispatching monitoring data.

BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the present invention, the drawings required for the present invention will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0056] Figure 1 is a schematic diagram of the framework process of the anomaly detection method for power dispatching monitoring data based on spatio-temporal anomaly Transformer proposed by the present invention;

[0057] Figure 2 is a schematic diagram of the spatio-temporal anomaly Transformer model;

[0058] Figure 3 is a schematic diagram of multi-head time self-attention;

[0059] Figure 4 is a schematic diagram of multi-head space self-attention;

[0060] Figure 5 is a schematic diagram of multi-head time anomaly self-attention;

[0061] Figure 6 is a schematic diagram of multi-head space anomaly self-attention;

[0062] Figure 7 It is a schematic diagram of the input data and output results of the algorithm of the present invention.

Specific Embodiments

[0063] For a better understanding of the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings.

[0064] It should be clear that the described embodiments of the invention are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative work belong to the scope of protection of the present invention.

[0065] The present invention provides a method for detecting anomalies in power dispatching monitoring data based on spatio-temporal anomaly Transformer. To meet the anomaly detection of power dispatching monitoring data, the present invention uses a time-space branch and a space-time branch to more fully extract the temporal dimension semantic information and spatial dimension semantic information in the monitoring data, and combines the reconstruction error, the temporal dimension correlation deviation, and the spatial dimension correlation deviation to calculate the anomaly score to judge anomalies.

[0066] Figure 1 It is a schematic diagram of the framework process of the method for detecting anomalies in power dispatching monitoring data based on spatio-temporal anomaly Transformer proposed by the present invention. The method includes the following steps:

[0067] Step 101, construct a spatio-temporal anomaly Transformer model, use the historical data of power dispatching monitoring as the input of the model, and calculate the reconstruction error, the temporal dimension correlation deviation, and the spatial dimension correlation deviation of the input data. Specifically:

[0068] Use the historical data of the operating conditions of the hardware and processes related to the power dispatching system collected by the power dispatching monitoring system as the training set of the model, denoted as where represents the set of real numbers, N represents the features of the data, specifically including the CPU usage rate, memory usage rate, disk I / O, network I / O of the server, and the CPU occupancy rate, memory occupancy rate, disk I / O, network I / O of each related process. T train is the total number of data in the historical data set χ train ;

[0069] Use a sliding window with a length of L and a step size of O to divide the historical data set χ train into multiple time windows, and the values of L and O are both 100. Denote the time window of the last data as x t as X t ={xt-L+1 , x t-L+2 ,..., x t}. Build a spatio-temporal anomaly Transformer model and calculate the reconstruction error, temporal dimension correlation deviation, and spatial dimension correlation deviation of the input data:

[0070] The spatio-temporal anomaly Transformer model consists of two parts: a space-time branch and a time-space branch. The space-time branch is formed by connecting a spatial Transformer and a temporal anomaly Transformer end to end, and the time-space branch is formed by connecting a temporal Transformer and a spatial anomaly Transformer end to end.

[0071] The spatial Transformer in the space-time branch is composed of M spatial Transformer layers, where the value of M is 3. Each spatial Transformer layer is composed of multi-head spatial self-attention, residual connection, two normalization layers, and a feed-forward network. Denote the number of heads of the multi-head spatial self-attention as H, and the value of H is 8. The calculation process of the h-th spatial self-attention head in the m-th spatial Transformer layer is as follows:

[0072]

[0073]

[0074] Z m,h = S m,h V m,h

[0075] Among them, is the output of the previous spatial Transformer layer, and the input of the first spatial Transformer layer is the transpose of the time window X t , and are three different linear layers, Q m,h , K m,h and V m,h are the query, key, and value respectively, d ST is the dimension of the query, key, and value, d ST takes the value of represents rounding down. Softmax(·) is the normalized exponential function, S m,h is the attention weight, and Z m,h is the output of this spatial self-attention head.

[0076] The output of the m-th spatial Transformer layer combines the outputs of all H spatial self-attention heads and passes through residual connections, two normalization layers, and a feed-forward network to obtain the output of this spatial Transformer layer. The process is as follows:

[0077]

[0078]

[0079]

[0080] Among them, Concat(·) represents concatenation, is a linear layer, Z m is the output of the multi-head spatial self-attention, LayerNorm(·) represents normalization, is the output of the first normalization layer, FFN(·) represents the feed-forward network, is the output of this spatial Transformer layer.

[0081] The temporal anomaly Transformer in the spatio-temporal branch consists of M temporal anomaly Transformer layers. Each temporal anomaly Transformer layer consists of multi-head temporal anomaly self-attention, residual connections, two normalization layers, and a feed-forward network. The number of heads of the multi-head temporal anomaly self-attention is also H, and the dimensions of the query, key, and value are d TAT d TAT takes the value of The calculation process of the temporal anomaly Transformer is basically the same as that of the spatial Transformer. On this basis, the temporal anomaly Transformer needs to additionally calculate the temporal dimension correlation deviation. The process is as follows:

[0082]

[0083]

[0084]

[0085] Among them, is the output of the previous temporal anomaly Transformer layer. The input of the first temporal anomaly Transformer layer is the transpose of the output of the spatial Transformer, is a linear layer, σ m,h when calculating the prior correlation of the temporal dimension, is the variance corresponding to the i-th data in the time window. Rescale(·) represents scaling the input to a sum of 1 by dividing by the sum, Exp(·) represents calculating the exponential with base e, i and j represent the i-th and j-th data in the time window, P m,h is the prior correlation in the time dimension, P i m,h is the prior correlation in the time dimension corresponding to the i-th data in the time window. is the attention weight corresponding to the i-th data in the time window. KL(·) represents calculating the KL divergence, TAD(X t ) is the time dimension correlation deviation of the time window X t .

[0086] The time Transformer in the time-space branch consists of M time Transformer layers. Each time Transformer layer consists of multi-head time self-attention, residual connection, two normalization layers, and a feed-forward network. The number of heads of the multi-head time self-attention is also H, and the dimensions of the query, key, and value are d TT , d TT takes the value of The calculation method of the time Transformer is the same as that of the space Transformer. The input of the first time Transformer layer is the time window X t .

[0087] The spatial anomaly Transformer in the time-space branch consists of M spatial anomaly Transformer layers. Each spatial anomaly Transformer layer consists of multi-head spatial anomaly self-attention, residual connection, two normalization layers, and a feed-forward network. The number of heads of the multi-head spatial anomaly self-attention is also H, and the dimensions of the query, key, and value are d SAT , d SAT takes the value of The calculation process of the spatial anomaly Transformer is basically the same as that of the space Transformer. The input of the first spatial anomaly Transformer layer is the transpose of the output of the time Transformer. On this basis, the spatial anomaly Transformer needs to additionally calculate the spatial dimension correlation deviation, and the process is as follows:

[0088]

[0089] Among them, C m,h is the spatial dimension prior correlation of the h-th spatial anomaly self-attention head in the m-th spatial anomaly Transformer layer, C m,h is a randomly initialized matrix. is the prior spatial dimensional correlation of the i-th feature with respect to the j-th feature, is the attention weight of the i-th feature with respect to the j-th feature, SAD(X t ) is the spatial dimensional correlation deviation of the time window X t .

[0090] The model combines the outputs of the time anomaly Transformer and the spatial anomaly Transformer and reconstructs the time window X through a linear layer t and calculates the reconstruction error, and the process is as follows:

[0091]

[0092]

[0093]

[0094] Among them, is the output of the time anomaly Transformer, is the transpose of the output of the spatial anomaly Transformer, X CAT is the result after splicing the two outputs, W RCO is the linear layer used during reconstruction, Sigmoid(·) represents the Sigmoid activation function, is the reconstruction of the time window X t , X t,n is the n-th feature in the time window X t , is the n-th feature in the reconstructed time window , is the reconstruction error.

[0095] Step 102, calculate the loss of the model by combining the reconstruction error, the time dimensional correlation deviation, and the spatial dimensional correlation deviation, and use the loss as the optimization objective to train the model, specifically:

[0096] Based on the reconstruction error obtained in Step 101, the time dimensional correlation deviation TAD(X t ) and the spatial dimensional correlation deviation SAD(X t ), calculate the loss of the spatio-temporal anomaly Transformer model, and the process is as follows:

[0097]

[0098]

[0099] Among them, k1, k2, and k3 are the coefficients of each item, and the values of k1, k2, and k3 are 10, 10, and 0.1 respectively. L max (X t ) is the loss of the model in the maximization stage during training, and L min (X t ) is the loss of the model in the minimization stage during training.

[0100] Use the Adam optimizer to train the network parameters and the spatial dimension prior correlation C in the spatio-temporal anomaly Transformer model according to the optimization objective m,h . The number of training iterations is E, the batch size is B, and the learning rate is lr. The value of E is 500, the value of B is 32, and the value of lr is 0.0001. Each iteration during training is divided into two parts: the maximization stage and the minimization stage. In the maximization stage, the optimization objective of the model is:

[0101]

[0102] In the minimization stage, the optimization objective of the model is:

[0103]

[0104] During the training process, if L max (X t ) and L min (X t ) do not decrease within E lr iterations, then the learning rate lr is reduced to half of the original. If L max (X t ) and L min (X t ) do not decrease within E es iterations, then the training process is terminated prematurely. The value of E lr is 3, and the value of E es is 10.

[0105] Step 103: Use the power dispatching monitoring data to be detected as the input of the model, and calculate the reconstruction error, the time dimension correlation deviation, and the spatial dimension correlation deviation. Specifically:

[0106] Denote the power dispatching monitoring data to be detected as x test The features in are the same as those in the power dispatching monitoring historical data χ train . Combine x test and the power dispatching monitoring data collected at the previous L - 1 time points to form a time window X test , and use X testInput it into the trained spatio-temporal anomaly Transformer model, and repeat step 101 to calculate X test 's reconstruction error Time dimension correlation deviation TAD(X test ) and spatial dimension correlation deviation SAD(X test ).

[0107] Step 104: Combine the reconstruction error, time dimension correlation deviation, and spatial dimension correlation deviation to calculate the anomaly score of the input data, and accordingly determine whether the input power dispatching monitoring data is abnormal. Specifically:

[0108] Based on the time window X test 's reconstruction error Time dimension correlation deviation TAD(X test ) and spatial dimension correlation deviation SAD(X test ), calculate the anomaly score of each data in X test . The process is as follows:

[0109]

[0110] Among them, ⊙ represents element-wise multiplication of matrices, λ is the coefficient of the time dimension correlation deviation TAD(X test ), and the value of λ is 0.01. Score(X test ) is the anomaly score of each data in the time window X test . The last anomaly score Score(x test ) in Score(X test ) is the anomaly score corresponding to the data x test to be detected.

[0111] Repeat step 104 to calculate the anomaly scores Score(χ train ) of all historical power dispatching monitoring data χ train . Take the maximum value in Score(χ train ) as the threshold thr. If the anomaly score Score(x test ) of the data x test to be detected is greater than thr, then the data x test to be detected is abnormal, realizing the anomaly detection of power dispatching monitoring data.

[0112] Figure 2It is a schematic diagram of the spatio-temporal anomaly Transformer model, which consists of a space-time branch and a time-space branch. The space-time branch is formed by connecting a spatial Transformer and a temporal anomaly Transformer end to end, and the time-space branch is formed by connecting a temporal Transformer and a spatial anomaly Transformer end to end. Each Transformer consists of corresponding multi-head self-attention, residual connections, two normalization layers, and a feed-forward network.

[0113] Figure 3 It is a schematic diagram of multi-head temporal self-attention. Each temporal self-attention head first maps the input to queries, keys, and values through different linear layers, then calculates the attention weights based on the queries and keys, and finally calculates the output of this temporal self-attention head based on the attention weights and values. The multi-head temporal self-attention concatenates the outputs of all temporal self-attention heads and maps them through a linear layer to obtain the final output.

[0114] Figure 4 It is a schematic diagram of multi-head spatial self-attention. The structure of multi-head spatial self-attention is the same as that of multi-head temporal self-attention. The difference between the two is that multi-head temporal self-attention calculates the attention weights between each time point in the input window, while multi-head spatial self-attention calculates the attention weights between each feature in the input window.

[0115] Figure 5 It is a schematic diagram of multi-head temporal anomaly self-attention. Multi-head temporal anomaly self-attention maps the input to the variance of the prior correlation in the time dimension through an additional linear layer on the basis of multi-head temporal self-attention and calculates the prior correlation in the time dimension accordingly. Multi-head temporal anomaly self-attention calculates the KL divergence between the prior correlation in the time dimension and the attention weights as the correlation deviation in the time dimension.

[0116] Figure 6 It is a schematic diagram of multi-head spatial anomaly self-attention. Multi-head spatial anomaly self-attention adds an additional learnable matrix initialized randomly as the prior correlation in the spatial dimension on the basis of multi-head spatial self-attention. Multi-head spatial anomaly self-attention calculates the distance between the prior correlation in the spatial dimension and the attention weights as the correlation deviation in the spatial dimension.

[0117] Figure 7It is a schematic diagram of the input data and output results of the algorithm of the present invention. The input of the algorithm of the present invention is the operation status monitoring data of the hardware and processes related to the power dispatching system business, including the CPU usage rate, memory usage rate, disk I / O, network I / O of the server, and the CPU occupancy rate, memory occupancy rate, disk I / O, network I / O of each related process. The output of the algorithm of the present invention is the anomaly score of the data to be detected, and the data to be detected with an anomaly score greater than the threshold is determined as abnormal data.

[0118] Algorithm 1 is the pseudocode for the process of determining whether the data to be detected is abnormal in the anomaly detection method for power dispatching monitoring data based on spatio-temporal anomaly Transformer:

[0119]

[0120]

[0121] For specific embodiments, 5 publicly available datasets are used for testing. The datasets are the monitoring data of multiple different complex physical devices or large facilities. The specific information of each dataset is shown in Table 1. To reduce the randomness of the results, all results are the average of 5 independent runs.

[0122] Table 1 Datasets used in specific embodiments

[0123] Dataset Length of training set Length of test set Number of features Abnormal ratio SMD 708,405 708,420 38 4.16％ MSL 58,317 73,729 55 10.53％ SMAP 135,183 427,617 25 12.79％ SWaT 495,000 449,919 51 12.14％ PSM 132,481 87,841 25 27.76％

[0124] As shown in Table 2, to verify the effectiveness of the proposed algorithm, 13 typical unsupervised anomaly detection algorithms are used as comparison algorithms in the embodiments of the present invention.

[0125] Table 2 Algorithms for comparison in specific embodiments

[0126] Method Publication and year LOF ACM SIGMOD, 2000 OCSVM Neural Computation, 2001 IF ICDM, 2008 MSCRED AAAI, 2019 BeatGAN IJCAI, 2019 USAD KDD, 2020 InterFusion KDD, 2021 GDN AAAI, 2021 GTA IEEE Internet of Things Journal, 2022 TranAD VLDB, 2022 AT ICLR, 2022 CAE-AD Information Sciences, 2022 DATN Electronics, 2023

[0127] The embodiments of the present invention use AUC to measure the results of anomaly detection. AUC is the area under the ROC curve. The ROC curve is a curve drawn by the accuracy rate and false alarm rate as the decision threshold changes. The minimum value of AUC is 0, and the maximum value is 1. The higher the accuracy rate of anomaly detection, the closer the value of AUC is to 1; the lower the accuracy rate of anomaly detection, the closer the value of AUC is to 0.

[0128] The AUC results of the embodiments of the present invention and other comparison methods on the public dataset are shown in Table 3. The bold font indicates the highest average AUC and the lowest average ranking. The embodiments of the present invention are represented by STAT in the table. It can be seen that the method for anomaly detection of power dispatching monitoring data based on spatio-temporal anomaly Transformer of the present invention has obtained the highest average AUC and the lowest average ranking on the public dataset, indicating that the anomaly detection accuracy of the embodiments of the present invention is high.

[0129] Table 3 AUC results compared with other comparison methods on the public dataset

[0130] SMD MSL SMAP SWAT PSM Average AUC Average ranking LOF 0.6197 0.5607 0.6302 0.7846 0.7289 0.6648 7.00 OCSVM 0.6066 0.5269 0.4071 0.7925 0.6383 0.5943 11.60 IF 0.6698 0.5980 0.5893 0.8374 0.6991 0.6787 4.80 MSCRED 0.6943 0.6242 0.3823 0.4835 0.7370 0.5843 8.40 BeatGAN 0.7171 0.6215 0.5228 0.7878 0.7370 0.6773 4.80 USAD 0.5918 0.5864 0.4851 0.7967 0.6436 0.6207 9.20 InterFusion 0.6826 0.5718 0.4425 0.7038 0.6952 0.6192 9.00 GDN 0.6366 0.5454 0.4705 0.6857 0.6933 0.6063 10.20 GTA 0.7186 0.6032 0.4892 0.5959 0.7727 0.6359 5.80 TranAD 0.6146 0.5871 0.5711 0.4806 0.6835 0.5874 9.40 AT 0.4973 0.4625 0.5091 0.5553 0.4146 0.4878 12.60 CAE-AD 0.7370 0.5659 0.4476 0.8183 0.7467 0.6631 5.40 DATN 0.7297 0.6494 0.4350 0.8233 0.7244 0.6724 5.00 STAT 0.7262 0.5529 0.5773 0.8092 0.7834 0.6898 4.40

[0131] Similarly, the embodiments of the present invention are applied to the SCADA application service dataset of the power dispatching automation system. This dataset records the operating conditions of the processes and hardware that support the operation of the SCADA application service of the power dispatching automation system within 24 hours. The dataset contains a total of 2,500 data. Among them, the first 1,000 data are used as historical data to train the spatio-temporal anomaly Transformer model, and the latter 1,500 data are used to verify the performance of the embodiments of the present invention.

[0132] Table 4 shows the AUC results of the embodiments of the present invention and other comparison methods on the SCADA application service dataset of the power dispatching automation system. The embodiments of the present invention are represented by STAT in the table. It can be seen that compared with all the selected comparison methods, the method for anomaly detection of power dispatching monitoring data based on spatio-temporal anomaly Transformer of the present invention has obtained the best AUC result. The comparison results on the actual power dispatching monitoring dataset combined with the comparison results on the public dataset show that when dealing with power dispatching monitoring data with a high dimensionality of attribute features and mutual correlation between multiple features, the present invention can effectively improve the anomaly detection accuracy compared with typical unsupervised anomaly detection methods.

[0133] Table 4 AUC results compared with other comparison methods on the SCADA application service dataset of the power dispatching automation system

[0134] Method LOF OCSVM IF MSCRED BeatGAN USAD InterFusion AUC value 0.6653 0.6712 0.6054 0.5951 0.6550 0.6693 0.6435 Method GDN GTA TranAD AT CAE-AD DATN STAT AUC value 0.6796 0.6449 0.6656 0.5692 0.6778 0.6461 0.6845

[0135] In summary, the embodiments of the present invention have the following beneficial effects:

[0136] In the technical solution implemented by the present invention, a spatio-temporal anomaly Transformer model is constructed, and historical power dispatching and monitoring data is used as the input of the model to calculate the reconstruction error, the time-dimensional correlation deviation, and the space-dimensional correlation deviation; the loss of the model is calculated by combining the reconstruction error, the time-dimensional correlation deviation, and the space-dimensional correlation deviation, and the model is trained with the loss as the optimization objective; the power dispatching and monitoring data to be detected is used as the input of the model to calculate the reconstruction error, the time-dimensional correlation deviation, and the space-dimensional correlation deviation; the anomaly score of the input data is calculated by combining the reconstruction error, the time-dimensional correlation deviation, and the space-dimensional correlation deviation, and it is determined whether the data is abnormal. According to the technical solution provided by the embodiments of the present invention, when facing the power dispatching and monitoring data with a high dimensionality of attribute features and mutual correlation among multiple features, compared with the typical unsupervised anomaly detection method, this method can improve the accuracy of anomaly detection of power dispatching and monitoring data.

[0137] The foregoing are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for detecting anomalies in power dispatching monitoring data based on spatio-temporal anomaly Transformer, characterized in that, It includes the following steps: (1) Construct a spatio-temporal anomaly Transformer model, use the historical data of power dispatching monitoring as the input of the model, and calculate the reconstruction error, the time-dimensional correlation deviation, and the space-dimensional correlation deviation. Specifically: Taking the historical data of the operating conditions of the hardware and processes related to the power dispatching system collected by the power dispatching monitoring system as the training set of the model, denoted as where represents the set of real numbers, N represents the features of the data, specifically including the CPU usage rate, memory usage rate, disk IO, network IO of the server, and the CPU occupancy rate, memory occupancy rate, disk IO, network IO of each related process; T train is the total number of data in the historical data set χ train ; Divide the historical dataset χ into multiple time windows using a sliding window of length L and step size O, and denote the last data as x train The time window of t is X t ={x t-L+1 , x t-L+2 ,..., x t}; Build a spatio-temporal anomaly Transformer model and calculate the reconstruction error, time-dimensional correlation deviation, and space-dimensional correlation deviation of the input data: The spatio-temporal anomaly Transformer model consists of two parts: a space-time branch and a time-space branch; the space-time branch is composed of a spatial Transformer and a time anomaly Transformer connected end to end, and the time-space branch is composed of a time Transformer and a spatial anomaly Transformer connected end to end; The spatial Transformer in the space-time branch consists of M spatial Transformer layers, and each spatial Transformer layer is composed of multi-head spatial self-attention, residual connection, two normalization layers, and a feed-forward network. Denote the number of heads of the multi-head spatial self-attention as H; the calculation process of the h-th spatial self-attention head in the m-th spatial Transformer layer is as follows: Z m,h = S m,h V m,h Among them, is the output of the previous spatial Transformer layer, and the input of the first spatial Transformer layer is the transpose of the time window X t . and are three different linear layers, where Q m,h , K m,h and V m,h are the query, key, and value respectively, d ST is the dimension of the query, key, and value, Softmax(·) is the normalization exponential function, S m,h is the attention weight, and Z m,h is the output of this spatial self-attention head; The m-th spatial Transformer layer combines the outputs of all H spatial self-attention heads and obtains the output of this spatial Transformer layer through residual connection, two normalization layers, and a feed-forward network. The process is as follows: Among them, Concat(·) represents concatenation, is a linear layer, and Z m is the output of the multi-head spatial self-attention, and LayerNorm(·) represents normalization, is the output of the first normalization layer, and FFN(·) represents the feed-forward network, is the output of this spatial Transformer layer; The temporal anomaly Transformer in the spatio-temporal branch consists of M temporal anomaly Transformer layers. Each temporal anomaly Transformer layer is composed of multi-head temporal anomaly self-attention, residual connections, two normalization layers, and a feed-forward network. The number of heads in the multi-head temporal anomaly self-attention is also H, and the dimensions of the query, key, and value are d TAT ; The calculation process of the temporal anomaly Transformer is the same as that of the spatial Transformer; The temporal anomaly Transformer needs to additionally calculate the temporal dimension correlation deviation, and the process is as follows: Among them, is the output of the previous time anomaly Transformer layer. The input of the first time anomaly Transformer layer is the transpose of the output of the spatial Transformer. is a linear layer, and σ m,h is the variance when calculating the prior correlation in the time dimension. is the variance corresponding to the i-th data in the time window. Rescale(·) means scaling the input to a sum of 1 by dividing by the sum, and Exp(·) means calculating the exponential with base e. i and j represent the i-th and j-th data in the time window, and P m,h is the prior correlation in the time dimension, and P i m,h is the prior correlation in the time dimension corresponding to the i-th data in the time window. is the attention weight corresponding to the i-th data in the time window. KL(·) means calculating the KL divergence, and TAD(X t ) is the time dimension correlation deviation of the time window X t ; The time Transformer in the time - space branch consists of M time Transformer layers. Each time Transformer layer is composed of multi - head time self - attention, residual connection, two normalization layers, and a feed - forward network. The number of heads of the multi - head time self - attention is also H, and the dimensions of the query, key, and value are d TT ; The calculation method of the time Transformer is the same as that of the space Transformer. The input of the first time Transformer layer is the time window X t ; The spatial anomaly Transformer in the time - space branch consists of M spatial anomaly Transformer layers. Each spatial anomaly Transformer layer is composed of multi - head spatial anomaly self - attention, residual connection, two normalization layers, and a feed - forward network. The number of heads of the multi - head spatial anomaly self - attention is also H, and the dimensions of the query, key, and value are d SAT ; The calculation process of the spatial anomaly Transformer is the same as that of the spatial Transformer. The input of the first spatial anomaly Transformer layer is the transpose of the output of the time Transformer; The spatial anomaly Transformer needs to additionally calculate the spatial - dimension correlation deviation, and the process is as follows: Among them, C m,h is the spatial dimension prior correlation of the h-th spatial anomaly self-attention head in the m-th spatial anomaly Transformer layer, is the spatial dimension prior correlation of the i-th feature to the j-th feature, is the attention weight of the i-th feature to the j-th feature, SAD(X t ) is the spatial dimension correlation deviation of the time window X t . The model combines the outputs of the temporal anomaly Transformer and the spatial anomaly Transformer and reconstructs the temporal window X through a linear layer t And calculates the reconstruction error, and the process is as follows: Among them, is the output of the time anomaly Transformer, is the transpose of the output of the space anomaly Transformer, X CAT is the result after concatenating the two outputs, W RCO is the linear layer used during reconstruction, and Sigmoid(·) represents the Sigmoid activation function, is the time window X t is the reconstruction of X t,n is the time window X t is the nth feature in X is the reconstructed time window is the nth feature in is the reconstruction error; (2) Calculate the loss of the model by combining the reconstruction error, the time-dimensional correlation deviation, and the space-dimensional correlation deviation, and use the loss as the optimization objective to train the model. Specifically: Based on the reconstruction error obtained in step (1) Time - dimension association deviation TAD(X t ) and space - dimension association deviation SAD(X t ), calculate the loss of the spatio - temporal anomaly Transformer model, and the process is as follows: where k1, k2, and k3 are the coefficients of each term, and L max (X t ) is the loss at the maximization stage during the training of the model, and L min (X t ) is the loss at the minimization stage during the training of the model; Use the Adam optimizer to train the network parameters and the spatial dimension prior correlation C in the spatio-temporal anomaly Transformer model according to the optimization objective m,h , the number of training iterations is E, the batch size is B, and the learning rate is lr; each iteration in training is divided into two parts: the maximization stage and the minimization stage. In the maximization stage, the optimization objective of the model is: In the minimization stage, the optimization objective of the model is: During the training process, if L max (X t ) and L min (X t ) do not decrease within E lr iterations, the learning rate lr is reduced to half of the original; if L max (X t ) and L min (X t ) do not decrease within E es iterations, the training process is terminated prematurely; (3) Use the power dispatching monitoring data to be detected as the input of the model, and calculate the reconstruction error, the time-dimensional correlation deviation, and the space-dimensional correlation deviation. Specifically: Let the power dispatching monitoring data to be detected be x test whose features are the same as those in the power dispatching monitoring historical data χ train ; Combine x test and the power dispatching monitoring data collected at its previous L - 1 time points to form a time window X test , and input X test into the trained spatio - temporal anomaly Transformer model, and repeat step (1) to calculate the reconstruction error test of X the time - dimension correlation deviation TAD(X test ) and the space - dimension correlation deviation SAD(X test ); (4) Calculate the anomaly score of the input data by combining the reconstruction error, the time-dimensional correlation deviation, and the space-dimensional correlation deviation, and determine whether the input power dispatching monitoring data is abnormal based on this. Specifically: Based on the time window X obtained in step (3) test the reconstruction error the time - dimension association deviation TAD(X test ), and the space - dimension association deviation SAD(X test ), calculate the anomaly score of each data in X test as follows: Among them, ⊙ represents element-wise multiplication between matrix elements, λ is the coefficient of the time dimension association deviation TAD(X test ), Score(X test ) is the anomaly score of each data in the time window X test ; the last anomaly score Score(x test ) in Score(X test ) is the anomaly score corresponding to the data x test to be detected; Repeat step (4) to calculate all historical power dispatching and monitoring data χ train for the anomaly score Score(χ train ). Take the maximum value in Score(χ train ) as the threshold thr. If the anomaly score Score(x test ) of the data x test to be detected is greater than thr, then the data x test to be detected is an anomaly, realizing the anomaly detection of power dispatching and monitoring data.

2. The method for detecting anomalies in power dispatching monitoring data based on spatio-temporal anomaly Transformer according to claim 1, wherein, In the step (1), the values of the length L and the step size O of the sliding window are both 100.

3. The abnormal detection method for power dispatching monitoring data based on spatio-temporal anomaly Transformer according to claim 1, characterized in that In the said step (1), the value of the number of layers M is 3, the value of the number of heads H is 8, d ST and d SAT take values of d TT and d TAT take values of denotes rounding down.

4. The abnormal detection method for power dispatching monitoring data based on spatio-temporal anomaly Transformer according to claim 1, wherein In the step (1), the spatial prior correlation C in the spatial anomaly Transformer m,h is a randomly initialized matrix.

5. The abnormal detection method for power dispatching monitoring data based on spatio-temporal anomaly Transformer according to claim 1, wherein In the step (2), the values of k1, k2, and k3 are 10, 10, and 0.1 respectively.

6. The abnormal detection method for power dispatching monitoring data based on spatio-temporal anomaly Transformer according to claim 1, characterized in that In the step (2), the value of the iteration number E is 500, the value of the batch size B is 32, the value of the learning rate lr is 0.0001, and the value of E lr is 3, and the value of E es is 10.

7. The method for abnormal detection of power dispatching monitoring data based on spatio-temporal anomaly Transformer according to claim 1, wherein In the step (4), the value of λ is 0.01.

Images