Time series data anomaly detection method and system based on flattened attention mechanism

Abstract

The present invention discloses a method and system for detecting anomalies of time series data based on a flattened attention mechanism, which belongs to the technical field of anomaly detection of time series data. The method comprises: obtaining time series data; dividing the time series data into time blocks to obtain embedding within the time block and embedding between time blocks; calculating the flattened attention value and self-attention value of each embedding; weighted fusion of the flattened attention value and self-attention value of each embedding to obtain the weighted fusion result of attention of each embedding; upsampling the weighted fusion result of attention of each embedding to obtain the upsampled result of each embedding; calculating the anomaly score of the time series data according to the upsampled result of each embedding; and determining the anomaly detection result of the time series data according to the anomaly score of the time series data. The accuracy of anomaly detection of time series data is improved, and the problem of low accuracy of anomaly detection of time series data is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data anomaly detection, and in particular to a time series data anomaly detection method and system based on a flattened attention mechanism. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In industrial control systems (ICS), a large number of sensors and actuators continuously generate multi-dimensional time series data, which contain important information about the system's operating status. Anomaly detection, as a key technology to ensure the safe and stable operation of industrial control systems, has always been a hot topic in academia and industry. At present, anomaly detection methods for industrial control system time series data are mainly divided into three categories: methods based on traditional statistical analysis, methods based on machine learning, and methods based on deep learning.

[0004] Traditional statistical analysis methods, such as principal component analysis (PCA) and autoregressive integrated moving average model (ARIMA), are computationally efficient but have difficulty capturing complex nonlinear relationships and have limited processing capabilities for high-dimensional data. Machine learning-based methods, such as support vector machines (SVM) and random forests (RF), have improved the accuracy of anomaly detection to a certain extent, but their performance and scalability are challenged when faced with large-scale, high-dimensional industrial control time series data.

[0005] In recent years, deep learning technology has shown great potential in the field of anomaly detection of industrial control time series data. Models such as long short-term memory network (LSTM) and convolutional neural network (CNN) can effectively capture the long-term dependencies and spatial characteristics of time series data. However, these methods still have some limitations: 1) high computational complexity, which makes it difficult to meet the real-time requirements of industrial control systems; 2) limited processing capabilities for long sequence data; 3) difficulty in effectively fusing multi-source heterogeneous data.

[0006] As an emerging self-supervised learning paradigm, contrastive learning has achieved remarkable results in fields such as computer vision and natural language processing. By learning the representation of data, contrastive learning makes the representation of similar samples closer and the representation of different samples more separated. This method has the potential to improve the model's sensitivity to abnormal patterns. However, the current anomaly detection methods based on the contrastive learning framework focus on traditional attention mechanism modeling. The high complexity of traditional attention makes it difficult for the model to have real-time detection capabilities, and does not consider the information integration of input data, which easily leads to the model's inability to fully learn the temporal information of the data; ultimately resulting in low accuracy in data anomaly detection. Summary of the invention

[0007] In order to solve the above problems, the present invention proposes a time series data anomaly detection method and system based on a flattened attention mechanism to improve the accuracy of time series data anomaly detection in industrial control systems.

[0008] To achieve the above object, the present invention adopts the following technical solution:

[0009] First, a time series data anomaly detection method based on a flattened attention mechanism is proposed, including:

[0010] Get time series data;

[0011] Divide the time series data into time blocks to obtain intra-time block embedding and inter-time block embedding;

[0012] For each embedding in the intra-time block embedding and the inter-time block embedding, the flattened attention value and the self-attention value of the embedding are calculated; the flattened attention value and the self-attention value of the embedding are weightedly fused to obtain the weighted fusion result of the embedded attention; the weighted fusion result of the embedded attention is upsampled to obtain the upsampled result of the embedded attention;

[0013] According to the up-sampled results embedded in the time block and the up-sampled results embedded between time blocks, the anomaly score of the time series data is calculated;

[0014] According to the anomaly score of the time series data, the anomaly detection result of the time series data is determined.

[0015] Furthermore, the process of calculating the embedded flattened attention value includes:

[0016] Compute the embedded query vector and key vector;

[0017] Merge the embedded query vector and the key vector to obtain an embedded merged vector;

[0018] Perform multi-layer perceptual transformation on the embedded merged vector to obtain the result after embedding multi-layer perceptual transformation;

[0019] According to the result after embedding multi-layer perceptual transformation, the embedded flattened attention value is calculated.

[0020] Furthermore, the embedded flattened attention value and the self-attention value are weightedly fused to obtain the embedded attention weighted fusion result, which includes:

[0021] Calculate the embedded value vector;

[0022] Perform multi-layer perceptual transformation on the embedded value vector to obtain the weight of the embedded flattened attention value;

[0023] According to the weight of the embedded flattened attention value, the weight of the embedded self-attention value is calculated;

[0024] According to the weight of the embedded flattened attention value and the weight of the self-attention value, the embedded flattened attention value and the self-attention value are weighted fused to obtain the embedded attention weighted fusion result.

[0025] Furthermore, according to the up-sampled result of the embedding in the time block, the anomaly score of the embedding in the time block is calculated;

[0026] According to the up-sampled result of the time block embedding, the abnormal score of the time block embedding is calculated;

[0027] The anomaly score embedded within a time block and the anomaly score embedded between time blocks are added together to obtain the anomaly score of the time series data.

[0028] Furthermore, the embeddedness within the time block and the embeddedness between the time blocks are processed by the trained data anomaly detection model to obtain the anomaly detection result of the time series data, wherein the data anomaly detection model adopts a contrastive learning framework, including a first branch and a second branch, the first branch processes the embeddedness within the time block to obtain the upsampled result of the embeddedness within the time block, and obtains the anomaly score of the embeddedness within the time block according to the upsampled result of the embeddedness within the time block; the second branch processes the embeddedness between the time blocks to obtain the upsampled result of the embeddedness between the time blocks, and obtains the anomaly score of the embeddedness between the time blocks according to the upsampled result of the embeddedness between the time blocks; the anomaly score of the embeddedness within the time block and the anomaly score of the embeddedness between the time blocks are added to obtain the anomaly score of the time series data; according to the anomaly score of the time series data, the anomaly detection result of the time series data is determined.

[0029] Furthermore, the loss function during data anomaly detection model training is equal to the difference between the second branch loss and the first branch loss divided by the length of the embedding between time blocks; the first branch loss and the second branch loss are the difference results of the anomaly score embedded between time blocks and the anomaly score embedded within the time block measured by relative entropy.

[0030] Furthermore, points with anomaly scores greater than or equal to a set threshold are determined as anomaly points of the time series data; wherein the set threshold is obtained by training a data anomaly detection model.

[0031] Secondly, a time series data anomaly detection system based on a flattened attention mechanism is proposed, including:

[0032] A data acquisition unit, used for acquiring time series data;

[0033] A data partitioning unit, used to partition the time series data into time blocks to obtain intra-time block embedding and inter-time block embedding;

[0034] The data anomaly detection unit is used to calculate the flattened attention value and self-attention value of each embedding in the embedding within the time block and the embedding between the time blocks; weightedly fuse the flattened attention value and the self-attention value of the embedding to obtain the weighted fusion result of the embedded attention; upsample the weighted fusion result of the embedded attention to obtain the upsampled result of the embedding; calculate the anomaly score of the time series data according to the upsampled result of the embedding within the time block and the upsampled result of the embedding between the time blocks; and determine the anomaly detection result of the time series data according to the anomaly score of the time series data.

[0035] Furthermore, the data anomaly detection unit is used to calculate the embedded query vector and key vector; merge the embedded query vector and key vector to obtain an embedded merged vector; perform a multi-layer perceptual transformation on the embedded merged vector to obtain an embedded multi-layer perceptual transformation result; and calculate an embedded flattened attention value based on the embedded multi-layer perceptual transformation result.

[0036] Furthermore, the data anomaly detection unit is used to calculate and obtain an embedded value vector; perform a multi-layer perceptual transformation on the embedded value vector to obtain the weight of the embedded flattened attention value; calculate the weight of the embedded self-attention value based on the weight of the embedded flattened attention value; and perform weighted fusion on the embedded flattened attention value and the self-attention value based on the weight of the embedded flattened attention value and the weight of the self-attention value to obtain an embedded attention weighted fusion result.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The present invention proposes a time series data anomaly detection method and system based on a flattened attention mechanism. The method divides the time series data into time blocks, and calculates the flattened attention value and self-attention value of each divided embedding, and captures the long-distance dependency of the embedding by calculating the flattened attention value; then, the flattened attention value and self-attention value of each embedding are weightedly fused to obtain the weighted fusion result of the attention of each embedding, and then the anomaly detection result of the time series data is obtained according to the weighted fusion results of the attention of the two embeddings; thereby improving the accuracy of anomaly detection of time series data.

[0039] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings in the specification, which constitute a part of the present application, are used to provide further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0041] Figure 1 A flow chart of a method for detecting anomalies in time series data based on a flattened attention mechanism disclosed in an embodiment;

[0042] Figure 2 The overall framework diagram of the time series data anomaly detection method based on the flattened attention mechanism disclosed in the embodiment;

[0043] Figure 3 Comparison of sequence data anomaly detection results disclosed in the embodiments. DETAILED DESCRIPTION

[0044] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0045] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0046] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0047] Example 1

[0048] Terminology explanation:

[0049] 1. Contrastive learning: It is a self-supervised learning method that aims to learn the representation of data so that similar samples are closer in the representation space, while different samples are farther away. In industrial control anomaly detection, contrastive learning enhances the model's ability to distinguish between normal and abnormal patterns by comparing the feature representations of normal data and abnormal data, thereby improving the accuracy and robustness of detection.

[0050] 2. Flattened attention mechanism: It is an improved form of attention mechanism, which reduces the computational complexity of the traditional attention mechanism by linearizing the calculation process. When processing long sequence data, the linear attention mechanism can efficiently capture long-distance dependencies while maintaining computational efficiency. This makes it particularly suitable for processing large-scale, long-time series data in industrial control systems, and effectively solves the limitations of traditional attention mechanisms in terms of computing resource consumption.

[0051] 3. Hybrid expert system: It is an integrated learning method consisting of multiple "expert" models and a "gating" network. Each expert model specializes in a specific type of input or task, while the gating network is responsible for dynamically selecting or combining the outputs of these experts. In the anomaly detection of industrial control time series data, the hybrid expert system allows different expert models to focus on different types of abnormal patterns, thereby improving the model's adaptability to diverse anomalies and detection accuracy.

[0052] 4. Patch: A small area extracted from the original data. In deep learning and computer vision tasks, a large image is divided into multiple small, overlapping or non-overlapping blocks called a patch; in time series data (time series data) analysis, a patch usually refers to a fixed-length data segment or time window extracted from a continuous time series.

[0053] This embodiment aims to solve the defects of the industrial control system time series data anomaly detection method, such as high data processing complexity, poor adaptive modeling ability, and inability to integrate multi-source heterogeneous data, as well as the problem that a single modeling method is difficult to adapt to the diverse types of anomalies in industrial control systems. At the same time, it is necessary to ensure the high accuracy and low false alarm rate of the anomaly detection model, and improve the timeliness and robustness of the industrial control system time series data anomaly detection model.

[0054] Taking into account the characteristics of time correlation between industrial control time series data, this embodiment proposes a time series data anomaly detection method based on a flattened attention mechanism. First, the industrial control time series data is sliced ​​into two different perspectives, in-patch and patch-wise, as two inputs of the data anomaly detection model built based on contrastive learning. At the same time, the flattened attention mechanism is combined to realize the modeling of the long-term dependency relationship of the multidimensional time series on the two inputs, and a designed hybrid expert system is used to fully integrate the representation results of the two inputs. Finally, the anomaly detection of the time series data is realized according to the difference between the two input representations. This embodiment can effectively realize the anomaly detection of industrial control system data, and can achieve the anomaly detection requirements with high accuracy and low false alarm rate.

[0055] The time series data anomaly detection method based on the flattened attention mechanism disclosed in this embodiment is described in detail.

[0056] The time series data anomaly detection method based on the flattened attention mechanism disclosed in this embodiment is as follows: Figure 1 , Figure 2 As shown, including:

[0057] Get time series data;

[0058] Divide the time series data into time blocks to obtain intra-time block embedding and inter-time block embedding;

[0059] For each embedding in the intra-time block embedding and the inter-time block embedding, the flattened attention value and the self-attention value of the embedding are calculated; the flattened attention value and the self-attention value of the embedding are weightedly fused to obtain the weighted fusion result of the embedded attention; the weighted fusion result of the embedded attention is upsampled to obtain the upsampled result of the embedded attention;

[0060] According to the up-sampled results embedded in the time block and the up-sampled results embedded between time blocks, the anomaly score of the time series data is calculated;

[0061] According to the anomaly score of the time series data, the anomaly detection result of the time series data is determined.

[0062] In this embodiment, before dividing the time series data into time blocks, normalization preprocessing is first performed. The specific method of normalization is shown in formula (I):

[0063] (I)

[0064] In formula (I), yes The mean of yes The standard deviation of After standardization ; Assume that in an industrial control system, define a data set express Devices and time steps The measured device values ​​obtained on the device include sensors and actuators. Indicates Devices, Yes Equipment In time The measured value at .

[0065] In order to obtain the input perspective of contrastive learning, the acquired time series is divided into time blocks of defined size. Divide into two perspectives: patch-wise and in-patch, obtain inter-patch data and in-patch data, and the data dimensions after division are: , Represents the dimension of the data within the time block, The dimension representing the data between time chunks.

[0066] Afterwards, the data within the time block and the data between time blocks are processed through the position encoding and value encoding in the Transformer to obtain the embedding within the time block and time block embedding .

[0067] The trained data anomaly detection model is used to embed the time block and time block embedding Processing is performed to obtain anomaly detection results of time series data, wherein the data anomaly detection model adopts a contrastive learning framework, including a first branch and a second branch, the first branch processes the embedding within the time block to obtain the upsampled result of the embedding within the time block, and obtains the anomaly score of the embedding within the time block according to the upsampled result of the embedding within the time block; the second branch processes the embedding between time blocks to obtain the upsampled result of the embedding between time blocks, and obtains the anomaly score of the embedding between time blocks according to the upsampled result of the embedding between time blocks; the anomaly score of the embedding within the time block and the anomaly score of the embedding between time blocks are added to obtain the anomaly score of the time series data; according to the anomaly score of the time series data, the anomaly detection result of the time series data is determined.

[0068] Among them, the first branch processes the embedding in the time block, and the process of obtaining the abnormality score embedded in the time block includes: calculating the flattened attention value and self-attention value embedded in the time block; weightedly fusing the flattened attention value and self-attention value embedded in the time block to obtain the weighted fusion result of the attention embedded in the time block; upsampling the weighted fusion result of the attention embedded in the time block to obtain the upsampled result embedded in the time block; and obtaining the abnormality score embedded in the time block according to the upsampled result embedded in the time block.

[0069] The second branch processes the embedding between time blocks, and the process of obtaining the abnormality score of the embedding between time blocks includes: calculating the flattened attention value and self-attention value of the embedding between time blocks; weightedly fusing the flattened attention value and self-attention value of the embedding between time blocks to obtain the weighted fusion result of the attention of the embedding between time blocks; upsampling the weighted fusion result of the attention of the embedding between time blocks to obtain the upsampled result of the embedding between time blocks; and obtaining the abnormality score of the embedding between time blocks based on the upsampled result of the embedding between time blocks.

[0070] Unlike traditional attention, flattened attention does not directly perform matrix multiplication, but merges the query vector with the value vector, and obtains a preliminary representation of sequence modeling after transformation by a multi-layer perceptron (MLP). For each embedding in the intra-time block embedding and the inter-time block embedding, the process of calculating the flattened attention value of the embedding includes:

[0071] (1) Calculate the embedded query vector and key vector, where ,in, They represent the query vector and key vector embedded in the time block respectively; Represent the query vector and key vector embedded between time blocks respectively; They are all initialization weight matrices.

[0072] (2) Merge the embedded query vector and key vector to obtain the embedded merged vector.

[0073] (II)

[0074] In formula (II), Respectively represent the merged vector embedded in the time block and the merged vector embedded between time blocks, Represents a merge operation.

[0075] (3) Perform a multi-layer perceptual transformation on the embedded merged vector to obtain the result after the embedded multi-layer perceptual transformation.

[0076] Will obtain Input to MLP is:

[0077] (III)

[0078] In formula (III), Represents the result of each layer of MLP, Represents the use of ReLu activation function, represents the weight matrix, Represents the bias vector.

[0079] (4) Based on the results of the embedded multi-layer perceptual transformation, the embedded flattened attention value is calculated.

[0080] Since the query vector and the key vector are merged previously, after the linear transformation of the MLP, the overall complexity of the calculation of the attention matrix depends on the dimension of the MLP hidden layer, which is more suitable for long time series in the field of industrial control. The calculation method of the flattened attention value is as follows:

[0081] (IV)

[0082] In formula (IV), They represent the flattened attention value embedded within a time block and the flattened attention value embedded between time blocks respectively; represents the Softmax activation function, Represents the operations involved in formula (III).

[0083] The process of weighted fusion of the embedded flattened attention value and the self-attention value to obtain the embedded attention weighted fusion result includes:

[0084] Calculate the embedded value vector;

[0085] Perform multi-layer perceptual transformation on the embedded value vector to obtain the weight of the embedded flattened attention value;

[0086] According to the weight of the embedded flattened attention value, the weight of the embedded self-attention value is calculated;

[0087] According to the weight of the embedded flattened attention value and the weight of the self-attention value, the embedded flattened attention value and the self-attention value are weighted fused to obtain the embedded attention weighted fusion result.

[0088] In this embodiment, the calculated flattened attention value of each embedding is input into the time block hybrid expert system. The hybrid expert system calculates the self-attention value of each embedding, and performs weighted fusion on the flattened attention value and self-attention value of each embedding. The self-attention weighted result is adjusted according to the importance of different features in the input data to obtain the weighted fusion result of attention of each embedding. Specifically, the flattened attention module is used to generate the importance weights related to the input data to control the weighted results in the self-attention mechanism, which is defined as follows: The specific method is shown in formula (V):

[0089] (V)

[0090] In formula (V), They are the weighted fusion results of the attention embedded in the time block and the weighted fusion results of the attention embedded between time blocks; Represents the Sigmod operation, The operations involved in formula (III) are: They represent the value vector embedded in the time block and the value vector embedded between time blocks respectively. The calculation process of the value vector is the same as The calculation process is the same as They represent the self-attention value embedded between time blocks and the self-attention value embedded within a time block respectively. The specific calculation method is shown in formula (VI):

[0091] (VI)

[0092] In formula (VI), represents the softmax normalization operation, represents the dimension of the predefined self-attention embedding vector, Represents the number of multi-head attention heads (head).

[0093] Afterwards, the output results processed by the time block hybrid expert model are upsampled and integrated into the same dimension to facilitate subsequent perspective difference comparison. The specific method of the upsampling operation is shown in formula (VII).

[0094] (VII)

[0095] In formula (VII), They are respectively the upsampling results after embedding between time blocks and the upsampling results after embedding within a time block. Represents an upsampling operation.

[0096] The loss function during training of the data anomaly detection model constructed in this embodiment is equal to the difference between the second branch loss and the first branch loss divided by the length of the embedding between time blocks; the first branch loss and the second branch loss are the difference results of the anomaly score embedded between time blocks and the anomaly score embedded within the time block measured by relative entropy.

[0097] The data anomaly detection model is trained by calculating the difference between embeddings within time blocks and embeddings between time blocks. Considering the rarity of anomalies and the common potential patterns shared by normal points, the representations of the same input should be similar. For normal samples, the representations learned from two different perspectives should be highly consistent, but the representations of abnormal samples are difficult to learn, resulting in significant differences in the representations learned from different perspectives. Therefore, the KL divergence (relative entropy) can be used to measure the difference between the two branches. The loss functions of the two branches are specifically expressed as follows:

[0098] (VIII)

[0099] In formula (VIII), Represent the first branch loss and the second branch loss respectively. represents the operation of computing the KLd divergence, Represents the detach() function, which is used to stop the gradient propagation. The purpose is to achieve alternating training and avoid the collapse of the contrastive learning model during training. By calculating the difference between the above two perspectives respectively, it can be used as the definition of the final overall loss function, as shown in formula (IX):

[0100] (IX)

[0101] In formula (IX), Represents the final loss function of the data anomaly detection model, Represents the embedded length value between time blocks after division.

[0102] This embodiment calculates the anomaly score embedded in the time block according to the up-sampled result embedded in the time block;

[0103] According to the up-sampled result of the time block embedding, the abnormal score of the time block embedding is calculated;

[0104] The anomaly score embedded within a time block and the anomaly score embedded between time blocks are added together to obtain the anomaly score of the time series data.

[0105] Points with anomaly scores greater than or equal to a set threshold are determined as anomalies of the time series data; wherein the set threshold is obtained by training the data anomaly detection model.

[0106] This embodiment designs an anomaly score and a set threshold for a data anomaly detection model, compares the two on a test set, and implements the anomaly detection function. The specific implementation process is as follows:

[0107] Given that normal time series data points usually have potential patterns and structures, there is a strong correlation between these data points, while abnormal points often deviate from this potential pattern and have a weak correlation with other data points. Therefore, the distance between the fusion results of different perspectives from normal points is smaller than the corresponding distance of abnormal points. Correspondingly, according to the above-obtained result after upsampling operation, , for the sensor data at each moment in the test set, the difference between the two input view distributions is calculated and summed to obtain the abnormal score of the point, as shown in formula (X):

[0108] (Ⅹ)

[0109] In formula (X), Represents the abnormal score of time series data, and the rest of the operations are similar to formula (VIII).

[0110] Set a priori thresholds based on the anomaly scores of the time series data during training. Count the anomaly scores point by point to get the anomaly score range: , Represents the minimum value of the abnormality score calculated for all points. represents the maximum value. Divide the anomaly score range into Equally spaced intervals ,in, Indicates the interval number, Then, the frequency of abnormal scores in each interval is counted to obtain a sufficiently detailed distribution description. The specific method is shown in formula (XI):

[0111] (XI)

[0112] In formula (XI), represents the number of occurrences of abnormal scores in the i-th interval, Represents each test point in the current interval.

[0113] The obtained frequencies can be used to calculate the frequency of each interval, namely: ,in, is the abnormal frequency, is the total time points.

[0114] By observing the frequency distribution histogram corresponding to the frequency results, two clusters are identified: normal cluster ; Abnormal cluster ,in, Represents the threshold of the normal cluster, which is used to determine whether a point belongs to a normal cluster. Represents the threshold of the abnormal cluster, which is used to determine whether a point belongs to an abnormal cluster. Represents the time step index threshold, which is used to exclude anomalies within a certain period of time and reduce false alarms.

[0115] Determine the number of time points for high-scoring clusters , , is the proportional coefficient selected according to the characteristics of the data set. In data with longer time steps, a smaller Furthermore, a fixed threshold is set : Sort all anomaly scores in the validation set in descending order, ,in, It is a list that stores the sorted abnormality scores of each point. Refers to descending sort operation. Select The maximum anomaly score is used as the threshold ,Right now , the overall operation is shown in formula (XII):

[0116] Formula (XII)

[0117] The anomaly score of the time series data is compared with the set threshold δ, and the points exceeding the threshold are considered to be anomalies. The specific method is shown in formula (XIII):

[0118] Formula (XIII)

[0119] in, Represents the decision result of the data anomaly detection model to determine whether an anomaly occurs at each time step. Respectively represent whether the decision result is abnormal or normal.

[0120] Existing methods for detecting anomaly in multi-sensor long-time-step time series data in industrial control systems often face multiple difficulties: on the one hand, they are often unable to cope with complex industrial control environments, resulting in inaccurate detection results and even a large number of false alarms; on the other hand, these methods are often oversimplified in the modeling process and fail to fully consider the multi-dimensional characteristics and dynamic variability of industrial control data. These limitations seriously restrict the effective application of anomaly detection technology in industrial control systems. Compared with the existing industrial control anomaly detection models, the beneficial effects of the method disclosed in this embodiment are as follows:

[0121] This embodiment can adaptively learn feature representations of different time scales through a comparative learning framework, solving the limitation that traditional methods can only process fixed time windows. This method can capture long-term and short-term abnormal patterns in industrial control systems and improve the flexibility and accuracy of detection.

[0122] This embodiment adopts a flattened attention mechanism to effectively handle the complex interactive relationship between multi-dimensional industrial control data. This mechanism not only takes into account the temporal dependency of data, but also captures the spatial correlation between different sensors, thereby comprehensively analyzing the temporal and spatial correlation of multi-dimensional data in the industrial control system and improving the accuracy of anomaly detection.

[0123] This embodiment introduces a time block hybrid expert system to achieve multi-angle and multi-level analysis of industrial control time series data. This method first analyzes the overall sequence pattern from a macro perspective, identifies possible abnormal sequences, and then deeply analyzes specific data features from a micro perspective. By combining the advantages of multiple "expert" models, this embodiment not only has a solid foundation in theory, but also shows excellent performance in practical applications, significantly improving the accuracy and reliability of anomaly detection.

[0124] In combination with actual scenarios, a widely used public dataset SWaT (Safe Water Treatment Dataset) is used to verify the effectiveness of the method disclosed in this embodiment.

[0125] On the SWaT dataset, the dimension of the MLP hidden layer is set to 256, the learning rate is 0.0001, the number of training rounds is 1, and the scaling factor is Set to 0.001. Figure 3 As shown in the figure, the test results of the data anomaly detection model disclosed in this embodiment for data anomaly detection are: accuracy (precision) is 93.95; recall (recall) is 99.77; F1 (F1-score) value is 96.77. Compared with the current advanced Transformer (Anomaly Transformer) detection algorithm, it has achieved a 5.1% improvement.

[0126] Example 2

[0127] In this embodiment, a time series data anomaly detection system based on a flattened attention mechanism is disclosed, including:

[0128] A data acquisition unit, used for acquiring time series data;

[0129] A data partitioning unit, used to partition the time series data into time blocks to obtain intra-time block embedding and inter-time block embedding;

[0130] The data anomaly detection unit is used to calculate the flattened attention value and self-attention value of each embedding in the embedding within the time block and the embedding between the time blocks; weightedly fuse the flattened attention value and the self-attention value of the embedding to obtain the weighted fusion result of the embedded attention; upsample the weighted fusion result of the embedded attention to obtain the upsampled result of the embedding; calculate the anomaly score of the time series data according to the upsampled result of the embedding within the time block and the upsampled result of the embedding between the time blocks; and determine the anomaly detection result of the time series data according to the anomaly score of the time series data.

[0131] Furthermore, the data anomaly detection unit is used to calculate the embedded query vector and key vector; merge the embedded query vector and key vector to obtain an embedded merged vector; perform a multi-layer perceptual transformation on the embedded merged vector to obtain an embedded multi-layer perceptual transformation result; and calculate an embedded flattened attention value based on the embedded multi-layer perceptual transformation result.

[0132] Furthermore, the data anomaly detection unit is used to calculate and obtain an embedded value vector; perform a multi-layer perceptual transformation on the embedded value vector to obtain the weight of the embedded flattened attention value; calculate the weight of the embedded self-attention value based on the weight of the embedded flattened attention value; and perform weighted fusion on the embedded flattened attention value and the self-attention value based on the weight of the embedded flattened attention value and the weight of the self-attention value to obtain an embedded attention weighted fusion result.

[0133] The method disclosed in Example 1 can be directly embodied as a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0134] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0135] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A time series data anomaly detection method based on a flattened attention mechanism, characterized in that: include: Obtain time series data from actuators and sensors in industrial control systems; The time series data of the actuators and sensors in the industrial control system are divided into time blocks to obtain intra-time block embedding and inter-time block embedding of the time series data of the actuators and sensors in the industrial control system; For each embedding in the intra-time block embedding and the inter-time block embedding, the flattened attention value and the self-attention value of the embedding are calculated; the flattened attention value and the self-attention value of the embedding are weightedly fused to obtain the weighted fusion result of the embedded attention; the weighted fusion result of the embedded attention is upsampled to obtain the upsampled result of the embedded attention; According to the up-sampled results embedded in the time block and the up-sampled results embedded between the time blocks, the anomaly scores of the time series data of the actuators and sensors in the industrial control system are calculated; Determine anomaly detection results of the time series data of the actuators and sensors in the industrial control system according to anomaly scores of the time series data of the actuators and sensors in the industrial control system; The process of calculating the flattened attention value of the embedding includes: Compute the embedded query vector and key vector; Merge the embedded query vector and the key vector to obtain an embedded merged vector; Perform multi-layer perceptual transformation on the embedded merged vector to obtain the result after embedding multi-layer perceptual transformation; According to the result after embedding multi-layer perception transformation, the embedded flattened attention value is calculated; The process of weighted fusion of the embedded flattened attention value and the self-attention value to obtain the embedded attention weighted fusion result includes: Calculate the embedded value vector; Perform multi-layer perceptual transformation on the embedded value vector to obtain the weight of the embedded flattened attention value; According to the weight of the embedded flattened attention value, the weight of the embedded self-attention value is calculated; According to the weight of the embedded flattened attention value and the weight of the self-attention value, the embedded flattened attention value and the self-attention value are weightedly fused to obtain the embedded attention weighted fusion result; The process of calculating the anomaly score of the time series data of the actuators and sensors in the industrial control system according to the up-sampled results embedded in the time block and the up-sampled results embedded between the time blocks includes: According to the up-sampled result of the embedding in the time block, the anomaly score of the embedding in the time block is calculated; According to the up-sampled result of the time block embedding, the abnormal score of the time block embedding is calculated; The anomaly scores embedded within a time block and the anomaly scores embedded between time blocks are added together to obtain the anomaly scores of the time series data of actuators and sensors in the industrial control system.

2. The method for detecting anomalies in time series data based on a flattened attention mechanism according to claim 1, characterized in that: The embedded data within a time block and the embedded data between time blocks are processed by a trained data anomaly detection model to obtain anomaly detection results of time series data of actuators and sensors in an industrial control system, wherein the data anomaly detection model adopts a contrastive learning framework, including a first branch and a second branch, the first branch processes the embedded data within the time block to obtain an upsampled result of the embedded data within the time block, and obtains an anomaly score of the embedded data within the time block according to the upsampled result of the embedded data within the time block; the second branch processes the embedded data between time blocks to obtain an upsampled result of the embedded data between time blocks, and obtains an anomaly score of the embedded data between time blocks according to the upsampled result of the embedded data between time blocks; the anomaly score of the embedded data within the time block and the anomaly score of the embedded data between time blocks are added to obtain an anomaly score of the time series data of the actuators and sensors in the industrial control system; according to the anomaly score of the time series data of the actuators and sensors in the industrial control system, the anomaly detection results of the time series data of the actuators and sensors in the industrial control system are determined.

3. The method for detecting anomalies in time series data based on a flattened attention mechanism according to claim 1, characterized in that: The loss function during data anomaly detection model training is equal to the difference between the second branch loss and the first branch loss divided by the length of the embedding between time blocks; the first branch loss and the second branch loss are the difference results of the anomaly score between the embedding between time blocks and the anomaly score within the time block measured by relative entropy.

4. The method for detecting anomalies in time series data based on a flattened attention mechanism according to claim 1, characterized in that: Points with anomaly scores greater than or equal to the set threshold are determined as anomalies in the time series data of actuators and sensors in the industrial control system.

5. A time series data anomaly detection system based on a flattened attention mechanism, characterized in that: include: A data acquisition unit, used to acquire time series data of actuators and sensors in the industrial control system; A data partitioning unit, used to partition the time series data of the actuators and sensors in the industrial control system into time blocks, and obtain intra-time block embedding and inter-time block embedding of the time series data of the actuators and sensors in the industrial control system; The data anomaly detection unit is used to calculate the flattened attention value and the self-attention value of each embedding in the intra-time block embedding and the inter-time block embedding; weightedly fuse the flattened attention value and the self-attention value of the embedding to obtain the weighted fusion result of the embedded attention; upsample the weighted fusion result of the embedded attention to obtain the upsampled result of the embedded; According to the up-sampled results embedded in the time block and the up-sampled results embedded between the time blocks, the anomaly scores of the time series data of the actuators and sensors in the industrial control system are calculated; according to the anomaly scores of the time series data of the actuators and sensors in the industrial control system, the anomaly detection results of the time series data of the actuators and sensors in the industrial control system are determined; The process of calculating the embedded flattened attention value includes: Compute the embedded query vector and key vector; Merge the embedded query vector and the key vector to obtain an embedded merged vector; Perform multi-layer perceptual transformation on the embedded merged vector to obtain the result after embedding multi-layer perceptual transformation; According to the result after embedding multi-layer perception transformation, the embedded flattened attention value is calculated; The process of weighted fusion of the embedded flattened attention value and the self-attention value to obtain the embedded attention weighted fusion result includes: Calculate the embedded value vector; Perform multi-layer perceptual transformation on the embedded value vector to obtain the weight of the embedded flattened attention value; According to the weight of the embedded flattened attention value, the weight of the embedded self-attention value is calculated; According to the weight of the embedded flattened attention value and the weight of the self-attention value, the embedded flattened attention value and the self-attention value are weightedly fused to obtain the embedded attention weighted fusion result; The process of calculating the anomaly score of the time series data of the actuators and sensors in the industrial control system according to the up-sampled results embedded in the time block and the up-sampled results embedded between the time blocks includes: According to the up-sampled result of the embedding in the time block, the anomaly score of the embedding in the time block is calculated; According to the up-sampled result of the time block embedding, the abnormal score of the time block embedding is calculated; The anomaly scores embedded within a time block and the anomaly scores embedded between time blocks are added together to obtain the anomaly scores of the time series data of actuators and sensors in the industrial control system.

Images