Bayesian network-based multivariate time sequence anomaly detection method
Through the Bayesian network-based multivariate time series anomaly detection method, the LSTM and Graphormer models are used to extract time and spatial features and calculate conditional probability, and the problem of anomaly detection in high-dimensional time series data is solved, achieving high accuracy and real-time performance Strong abnormality detection.
Patent Information
- Application Number
- CN202411427645.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-14
- Publication Date
- 2025-01-07
AI Technical Summary
The prior art is difficult to effectively detect abnormalities in high-dimensional time series data, especially in wireless communications and industrial applications. Traditional methods cannot effectively integrate the causal relationship between time information and spatial, resulting in high error rate of abnormal detection and inability to detect network problems in time.
Using a multivariate time series anomaly detection method based on Bayesian network, time historical features are extracted through LSTM neural network, and directed acyclic graph structure is generated by combining PC algorithms and Graphormer models. Bayesian formulas and conditional normalized flow calculation conditions are used. Probability, real-time detection of data abnormalities.
It improves the accuracy and universality of multivariate time series detection, can detect and dynamically update data abnormalities in real time, and promptly detect network problems, and is suitable for different types of industrial data.
Smart Images
Figure CN119272198A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data anomaly detection, and in particular to a multivariate time series anomaly detection method based on a Bayesian network. Background Art
[0002] High-dimensional time series data is becoming increasingly common in various fields such as wireless communications, industrial applications, and healthcare. However, due to its complex time dependency, high dimensionality, and scarcity of labels, detecting anomalies in such data is challenging. In this context, large-scale industrial sensor data requires effective data anomaly detection algorithms to remove application barriers for emerging data application technologies such as artificial intelligence, which is of great significance to improving industrial production efficiency and communication industry security.
[0003] In the era of fast-growing big data, data is often in time series, which means that the current value in the data is affected by the earlier value, rather than being independent. At the same time, the potential spatial causal relationship between different attribute elements using traditional methods based on variational autoencoders (VAE) using reconstruction errors as anomaly scores and deep autoencoder Gaussian mixture models (DAGMM) to detect density cannot effectively integrate the temporal information in the data with the spatial causal relationship between elements, resulting in a large error rate in anomaly detection and a poor maintenance of system operation. Summary of the invention
[0004] The present invention provides a multivariate time series anomaly detection method based on a Bayesian network. Compared with traditional methods, this method has higher accuracy and wider universality for multivariate time series. It can detect different types of industrial data by using a machine learning method, does not need to limit data characteristics, has strong real-time performance, can detect and dynamically update data anomalies in real time, and promptly discover network problems.
[0005] The embodiment of the present invention provides a multivariate time series anomaly detection method based on a Bayesian network, comprising the following steps:
[0006] Step 1: collect original industrial sensor time series data, clean and standardize the data to obtain multivariate time data;
[0007] Step 2, input the multivariate time data of each original industrial sensor into the LSTM neural network to preliminarily extract the time history characteristics of the original industrial sensor data;
[0008] Step 3, using the PC algorithm that integrates expert knowledge and the prior knowledge of domain experts to generate multivariate time series data into a directed acyclic graph structure;
[0009] Step 4: According to the Bayesian formula, the time history features and the directed acyclic graph structure are used to extract the time information and spatial information feature vectors in the fused data through the Graphormer neural network as the spatiotemporal feature information of the current time node data;
[0010] In step 5, the spatiotemporal characteristic information of the data at the current time node is used as a condition and input into the conditional normalized stream together with the multivariate time data in step 1. Through the reversible transformation of the normalized stream, the conditional probability of the data at the current moment is output. The conditional probability is used as the anomaly score of the system to reflect the degree of anomaly of the data.
[0011] Optionally, in one embodiment of the present invention, step 2 specifically includes:
[0012] Each node The multivariate time data is input into the LSTM network, and the time history information of each time step t is recursively extracted. In order to parameterize the conditional density, x i The superset pa(x i ) i：t Modeled as Modeled as Represents x i The historical information of the parent node before time t, Represents the historical information of node i before time t.
[0013] Optionally, in one embodiment of the present invention, step 3 specifically includes:
[0014] The network structure is initialized according to the prior knowledge of domain experts, and conditional independence tests are performed to gradually correct and improve the network structure. Specifically, in the initialization phase, the initial connection states of nodes and edges are set according to the prior information provided by experts. Through conditional independence tests, edge connections are gradually deleted or retained, and directions are given to the retained edges. During the conditional independence test, expert knowledge is given priority to ensure the consistency of the generated network structure in statistics and domain knowledge.
[0015] Optionally, in one embodiment of the present invention, step 4 specifically includes:
[0016] Graphormer is used to aggregate historical information, and layer normalization is applied before multi-head self-attention and feed-forward blocks. and Aggregate with the graph adjacency matrix A into a fixed-length representation in
[0017]
[0018] in, Represents all node features of the final iteration of Graphormer at time t, and each hidden state Corresponding to the representation of node i, Aggregation parameters representing the node's own information, Parameters that represent improved dependency properties.
[0019] Optionally, in one embodiment of the present invention, step 5 specifically includes:
[0020] Introduction of Graphormer polymerization is input to the flow normalization as the conditional information for density estimation, The conditional density of is written as:
[0021]
[0022] Among them, q(z) is selected as the standard normal distribution;
[0023] The log density of a multivariate time series X is:
[0024]
[0025] For each time series X i Generate a conditional density and used as a measure of anomaly for the constituent sequences.
[0026] The multivariate time series anomaly detection method based on Bayesian network in the embodiment of the present invention has high anomaly detection accuracy. Compared with traditional methods, this method has higher accuracy and wider universality for multivariate time series. The machine learning method can detect different types of industrial data without limiting data characteristics and has strong real-time performance. It can detect and dynamically update data anomalies in real time and discover network problems in a timely manner.
[0027] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0029] Figure 1 A flowchart of a multivariate time series anomaly detection method based on a Bayesian network according to an embodiment of the present invention;
[0030] Figure 2 It is a structural diagram of the LSTM network in the present invention;
[0031] Figure 3Schematic diagram of a directed acyclic graph in a Bayesian network. DETAILED DESCRIPTION
[0032] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0033] For industrial system big data (mainly wireless communication data), due to its complex data structure and massive data volume, if the system data is abnormal, it is difficult to accurately report the abnormality. In order to solve the technical difficulties of data anomaly detection in industrial systems, especially wireless communication networks, the present invention proposes a data anomaly detection method with high anomaly detection accuracy, wide universality, and strong real-time performance. Through the Bayesian network and Graphormer model related algorithms, the abnormal data in the industrial database and communication throughput data are cleaned, effectively improving industrial operation efficiency and communication data security. Figure 1 As shown, the multivariate time series anomaly detection method based on Bayesian network includes the following steps:
[0034] Step 1: Collect the original industrial sensor time series data, clean and standardize the data to obtain multivariate time data.
[0035] Step 2: Input the multivariate time data of each original industrial sensor into the LSTM neural network to preliminarily extract the time history features of the original industrial sensor data.
[0036] Based on the causal conditional probability factorization of Bayesian networks, which is the mathematical basis of the whole model, a multivariate time series decomposition method along the time and space dimensions is proposed to capture the mathematical representation of temporal dependencies in time series and spatial dependencies between different series.
[0037] According to the Bayesian property, for a multivariate time series X = (X 1 , X 2 , ..., X N ), can be obtained by multiplying the conditional probability of all nodes p(X i |pa(X i )) to calculate its probability density, where pa(X i ) means X i To further describe the dependencies in the time dimension, each conditional density can be divided along the time dimension. Specifically, at a given time point t, the variable Depends on its previous history and parent nodes in the DAG. This dependency can be expressed as:
[0038]
[0039] in, represents the history of node i before time t, pa(X i )={X j : A ij ≠0} means X i The parent node of .
[0040] LSTM temporal feature extraction is composed of a long short-term memory (LSTM) network neural network, which is a recurrent neural network (RNN) designed to capture long-term dependencies in sequence data. Unlike traditional RNNs, LSTM networks solve the gradient vanishing problem in long-term sequence learning by introducing memory units and a series of gating mechanisms that regulate the flow of information.
[0041] like Figure 2 As shown in Figure 1, an LSTM unit is mainly composed of a memory unit, an input gate, a forget gate, and an output gate. At time step t, the input vector of the LSTM hidden layer is The output vector is The memory unit is Among them, the forget gate:
[0042]
[0043] Input Gate:
[0044]
[0045] Output Gate:
[0046]
[0047] in, Indicates the corresponding multiplication of vector elements; W indicates weight; σ is the sigmoid function, and b is the bias term. Tanh is the hyperbolic tangent function. The memory unit and the gate unit are used to save historical information and long-term states. The forget gate enables LSTM to learn continuous tasks and reset the internal state.
[0048] Each node The time series data is input into the LSTM network, and the time history information of each time step is recursively extracted. At time step t, the input vector of the LSTM hidden layer is The output vector is To parameterize the conditional density, we can use x i The superset pa(x i ) i：t Modeled as Modeled as Through such modeling, the original formula can be rewritten as:
[0049]
[0050] That is to say, Represents x i Parent node pa(x i ) to the historical information before time t, Represents the historical information of node i before t.
[0051] Step 3: Use the Peter-Clark (PC) algorithm that integrates expert knowledge and the prior knowledge of domain experts to generate multivariate time series data into a directed acyclic graph structure, such as Figure 3 shown.
[0052] Bayesian network graph structure generation based on expert knowledge fusion algorithm. In Bayesian network structure learning, the PC algorithm combined with expert knowledge is an effective method. By integrating the prior knowledge of domain experts and data-driven conditional independence tests, the accuracy and rationality of the generated network structure can be improved. The PC algorithm and the prior knowledge of domain experts are used to generate multivariate time series data into a directed acyclic graph structure (DAG).
[0053] In Bayesian network structure learning, the PC algorithm combined with expert knowledge is an effective method. By integrating the prior knowledge of domain experts and data-driven conditional independence tests, it can improve the accuracy and rationality of the generated network structure.
[0054] In Bayesian network structure learning, PC algorithm is a classic and widely used method. The algorithm infers the dependency between variables from data through conditional independence test. However, relying solely on data-driven methods may result in the generated network structure being inconsistent with actual domain knowledge. In order to improve the accuracy and rationality of Bayesian network structure generation, the present invention proposes a Bayesian network graph structure generation method based on PC algorithm and integrating expert knowledge.
[0055] The method first initializes the network structure based on the prior knowledge of domain experts, and then uses data to perform conditional independence tests to gradually correct and improve the network structure. Specifically, in the initialization phase, the initial connection states of nodes and edges are set according to the prior information provided by experts. Then, the edge connections are gradually deleted or retained through conditional independence tests, while directions are given to the retained edges. In the process of conditional independence tests, expert knowledge is given priority to ensure the consistency of the generated network structure in statistics and domain knowledge.
[0056] Many experimental results show that compared with the traditional PC algorithm, the Bayesian network structure generation method that integrates expert knowledge performs better on multiple real data sets. The generated network structure not only has higher accuracy in data fitting, but also better reflects domain knowledge, enhancing the interpretability and application value of the model. By combining data-driven methods and expert prior knowledge, this method provides a more effective and practical solution for Bayesian network structure learning.
[0057] Step 4: According to the Bayesian formula, the time history features and the directed acyclic graph structure are used to extract the time information and spatial information feature vectors in the fused data through the Graphormer neural network as the spatiotemporal feature information of the current time node data.
[0058] Graphormer spatiotemporal feature extraction is used to learn fused feature representations of spatiotemporal data as conditional input to the conditional normalization flow.
[0059] Furthermore, when historical information and When the dimensions of do not match, it cannot be directly used for parameterization. Therefore, Graphormer is adopted to aggregate historical information. Graphormer is built on the original implementation of the classic transformer encoder. In addition, layer normalization (LN) is applied before multi-head self-attention (MHA) and feed-forward block (FFN). Historical information and Aggregate with the graph adjacency matrix A into a fixed-length representation in
[0060]
[0061] in, Represents all node features of the final iteration of Graphormer at time t, where each hidden state Corresponding to the representation of node i. Represents the aggregation parameter of the node's own information. Concat represents the concatenation operation of the matrix. ReLU is the abbreviation of Rectified Linear Unit, a common activation function. In addition, an additional transformation is introduced, namely To improve the representation of dependency properties, we can write:
[0062]
[0063] In step 5, the spatiotemporal feature information of the data at the current time node is used as a condition and input into the conditional normalized stream together with the multivariate time data in step 1. Through the reversible transformation of the normalized stream, the conditional probability of the data at the current moment is output. The conditional probability can be used as the abnormality score of the system to reflect the degree of abnormality of the data. The lower the conditional probability, the lower the possibility of the system occurring under normal circumstances, and the greater the abnormal probability. The abnormal probability exceeding the threshold is judged as abnormal, and the abnormal probability not exceeding the threshold is judged as normal.
[0064] The conditional normalization flow anomaly detection module is used to calculate the conditional probability density. During anomaly detection, the lower the density, the more likely it is that an anomaly exists.
[0065] Introduction of Graphormer polymerization is input to the flow normalization as the conditional information for density estimation. The conditional density can be written as:
[0066]
[0067] Among them, q(z) is selected as the standard normal distribution, so that the characteristics and trends of each data point can be better understood, and future development trends and behaviors can be predicted. In short, by combining Graphormer and normalized flow, time series data can be better processed, providing more accurate and effective solutions for fields such as time series analysis and forecasting. This method can better understand and analyze the complex relationships and patterns in the data set, ultimately leading to more effective time series data modeling and analysis, and obtaining the logarithmic density of the multivariate time series X:
[0068]
[0069] Furthermore, to detect abnormal behavior, it is assumed that anomalies deviate significantly from the majority of data instances and therefore exhibit low density. Therefore, the density calculated by the above formula is used as a method for anomaly detection, where a lower density indicates a greater likelihood of anomaly. In addition to evaluating the density of the entire dataset X, we also evaluate the density of each component sequence X. i Generate a conditional density and use it as an anomaly measure for the constituent sequences. A low density p(X) can be attributed to one or a few low conditional densities in the Bayesian network, indicating that the anomalous behavior can be traced back to a single sequence. In this way, anomalous behavior can be detected more comprehensively and further understood as to why the anomaly occurs.
[0070] The multivariate time series anomaly detection method based on Bayesian network of the present invention is described in detail below in conjunction with specific embodiments.
[0071] In order to evaluate the effectiveness of a multivariate time series anomaly detection method based on Bayesian networks in anomaly detection and density estimation, the present invention conducted a series of experiments and selected the SWaT public dataset for the experiments.
[0072] SWaT: The Safe Water Treatment (SWaT) dataset is from the Safe Water Treatment Testbed coordinated by the Singapore Public Utilities Board (Goh et al.
[68] , 2016). The data was collected from 77 sensors at a frequency of 1 second for 4 hours, collecting 15,000 samples. A total of 6 attacks were conducted, all of which were concentrated in the last 1.5 hours, resulting in about 13% of the time steps being considered abnormal. The data nodes with a variance of 0 in the SWaT dataset were deleted, and finally 44 valid nodes were obtained.
[0073] The performance of the proposed method is compared with the mainstream anomaly detection methods, including:
[0074] AE: This method consists of an encoder and a decoder to reconstruct the output from the representation.
[0075] VAE-LSTM: This method combines VAE and LSTM to model the underlying distribution of multidimensional data and reconstruct it using the desired distribution.
[0076] DAGMM: This method combines AE and GMM to perform anomaly detection through dimensionality reduction and density estimation.
[0077] GDN (Graph Deviation Network): This method combines graph neural networks (GNN) and deviation-based detection mechanisms to effectively capture complex dependencies in time series and identify anomalies.
[0078] MTAD-GAT (Multivariate Time-Series Anomaly Detection with GraphAttention Networks): This method combines the graph attention network (GAT) and the temporal attention network (TCN), which can effectively capture the spatiotemporal dependencies in time series data for anomaly detection.
[0079] Experiments were conducted on two different datasets using the above baselines and GBNCNF, and five metrics were measured: precision, recall, F1 score, accuracy, and area under the ROC curve. The specific experimental results are shown in Table 1. Bold fonts are used to show the best performance of each metric. In the SWaT dataset, we use F1 score and accuracy as key metrics. GBNCNF performs best with an accuracy of 93.5%.
[0080] Table 1 Experimental results
[0081]
[0082] According to the multivariate time series anomaly detection method based on Bayesian network proposed in the embodiment of the present invention, the original multivariate time series data is accessed, the time feature vector of each time series data is extracted, the spatial structure diagram of the data node is generated through the original multivariate time series data, the time feature of the time series data and the spatial structure information of the data node are integrated, the spatiotemporal fusion feature vector of the data is generated, the original multivariate time series and the spatiotemporal fusion feature vector as a condition are accessed, the conditional probability density of the data anomaly is generated, and the abnormal state of the system is evaluated. This method has higher accuracy and universality for multivariate time series. The machine learning method can detect different types of industrial data without limiting the characteristics of the data and has strong real-time performance. It can detect and dynamically update data anomalies in real time and discover network problems in a timely manner.
[0083] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0084] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0085] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.
Claims
1. A multivariate time series anomaly detection method based on Bayesian network, characterized in that: The following steps are involved: Step 1: collect original industrial sensor time series data, clean and standardize the data to obtain multivariate time data; Step 2, input the multivariate time data of each original industrial sensor into the LSTM neural network to preliminarily extract the time history characteristics of the original industrial sensor data; Step 3, using the PC algorithm that integrates expert knowledge and the prior knowledge of domain experts to generate multivariate time series data into a directed acyclic graph structure; Step 4: According to the Bayesian formula, the time history features and the directed acyclic graph structure are used to extract the time information and spatial information feature vectors in the fused data through the Graphormer neural network as the spatiotemporal feature information of the current time node data; In step 5, the spatiotemporal characteristic information of the data at the current time node is used as a condition and input into the conditional normalized stream together with the multivariate time data in step 1. Through the reversible transformation of the normalized stream, the conditional probability of the data at the current moment is output. The conditional probability is used as the anomaly score of the system to reflect the degree of anomaly of the data.
2. The method according to claim 1, characterized in that Step 2 specifically includes: Each node The multivariate time data is input into the LSTM network, and the time history information of each time step t is recursively extracted. In order to parameterize the conditional density, x i The superset pa(x i ) i：t Modeled as Modeled as Represents x i The historical information of the parent node before time t, Represents the historical information of node i before time t.
3. The method according to claim 1, characterized in that Step 3 specifically includes: The network structure is initialized according to the prior knowledge of domain experts, and conditional independence tests are performed to gradually correct and improve the network structure. Specifically, in the initialization phase, the initial connection states of nodes and edges are set according to the prior information provided by experts. Through conditional independence tests, edge connections are gradually deleted or retained, and directions are given to the retained edges. During the conditional independence test, expert knowledge is given priority to ensure the consistency of the generated network structure in statistics and domain knowledge.
4. The method according to claim 1, characterized in that: Step 4 specifically includes: Graphormer is used to aggregate historical information, and layer normalization is applied before multi-head self-attention and feed-forward blocks. and Aggregate with the graph adjacency matrix A into a fixed-length representation in in, Represents all node features of the final iteration of Graphormer at time t, and each hidden state Corresponding to the representation of node i, Aggregation parameters representing the node's own information, Parameters that represent improved dependency properties.
5. The method according to claim 1, characterized in that Step 5 specifically includes: Introduction of Graphormer polymerization is input to the flow normalization as the conditional information for density estimation, The conditional density of is written as: Among them, q(z) is chosen as the standard normal distribution; The log density of a multivariate time series X is: For each time series X i Generate a conditional density and used as a measure of anomaly for the constituent sequences.