Anomaly Detection Method for Distributed Systems Based on Log Workflow Mining

By constructing a workflow graph and using a graph convolutional neural network for anomaly detection, the problem of log parsing and detection in distributed systems is solved, achieving efficient anomaly detection and localization.

CN116560943BActive Publication Date: 2026-11-10XIDIAN UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202310519321.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-05-09
Publication Date
2026-11-10
Estimated Expiration
2043-05-09

AI Technical Summary

Technical Problem

Existing technologies struggle to effectively handle massive amounts of unstructured logs in distributed systems, leading to difficulties in anomaly detection and an inability to fully utilize the spatial characteristics between logs.

Method used

Event templates are extracted by parsing logs, time-series relationships are explored to construct a workflow graph, graph convolutional neural networks are used for anomaly detection, and Top-K pooling mechanism is combined to improve detection accuracy.

Benefits of technology

It improves the accuracy of anomaly detection in distributed systems, enabling the detection of various anomaly types and the location of anomalies, simplifies the problem of scattered log processing, and enhances system security.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116560943B_ABST
    Figure CN116560943B_ABST
Patent Text Reader

Abstract

This invention discloses a distributed system anomaly detection method based on log workflow mining, primarily addressing the difficulty of anomaly detection in massive unstructured distributed system logs. The solution includes: 1) parsing the logs generated by the distributed system, extracting event templates, and mining the temporal relationships between log events; 2) constructing a single-process FSM model for the system, and integrating the FSMs corresponding to each process using inter-process channels to obtain a workflow graph; 3) abstracting the workflow graph, checking it based on temporal relationships, and modifying parts that do not conform to the temporal relationships between instances to obtain the final workflow graph model; 4) using a graph convolutional neural network model based on a Top-K pooling mechanism to perform graph classification on the final workflow graph to identify abnormal workflow graphs. This invention can achieve anomaly detection for massive unstructured logs of distributed systems and effectively improves the detection accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of information security technology, and further relates to anomaly detection methods, specifically a distributed system anomaly detection method based on log workflow mining, which can be used for anomaly detection and identification in mainstream distributed systems. Background Technology

[0002] Distributed systems can dynamically distribute tasks and manage them globally, possessing high fault tolerance, scalability, and resource sharing capabilities. They are widely used in cloud computing platforms and serve numerous big data application platforms. The increasing computational demands have led to a significant increase in the number of computer nodes coordinating tasks in distributed systems. Furthermore, the complex system structure and functions of distributed systems directly result in a significant increase in the probability of anomalies and failures. Once an anomaly occurs, it can cause substantial losses to service providers and customers. The following characteristics of distributed systems are common causes of anomalies: ① Diverse distributed communication protocols and massive computational demands. Distributed systems process massive tasks in parallel, and the enormous computational demands lead to diverse communication methods, placing immense pressure on the data processing capabilities of the distributed system. This results in communication errors and data loss between nodes. ② A large number of computing nodes in distributed systems. The probability of failure increases when a large number of computing nodes are executed during complex tasks, manifesting as server node downtime and "zombie" phenomena. ③ Complex distributed system architecture and a wide variety of components. Complex functions lead to complex architectures in distributed systems, requiring numerous components to support corresponding distributed services and resource sharing among a large number of computing nodes. If security vulnerabilities exist between components or nodes, they are very easy to be attacked by hackers, leading to the paralysis of distributed systems and data leaks.

[0003] Anomaly detection technology is a crucial step in ensuring system security. Among most anomaly detection techniques, system logs are widely used because they record vital data such as system execution flow, system status information, and records of various emergencies, providing a valuable source of information for anomaly detection. Current research methods aim to bridge the gap between low-level logs and higher-level human-understandable diagrams and language by constructing workflows to model system operation, with system logs serving as a key data source for workflow construction. However, analyzing distributed system logs for anomaly detection still faces several challenges. First, distributed systems generate massive amounts of logs, sometimes reaching gigabytes per hour. Second, logs generated by different components within a distributed system involve diverse information, resulting in varied unstructured logs. Furthermore, the resource-sharing nature of distributed systems leads to scattered and difficult-to-process logs, all of which pose significant difficulties for anomaly detection. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a distributed system anomaly detection method based on log workflow mining. This method solves the problems of difficulty in parsing and utilizing massive amounts of unstructured logs, and the inability to utilize the spatial characteristics existing between logs for anomaly detection. This invention can utilize workflow graphs to mine the information spatial structure between distributed system logs, effectively improving the accuracy of distributed system log anomaly detection.

[0005] The approach to implementing this invention is as follows: First, extract event templates from logs through log parsing and mine the temporal relationships between log events; then, construct a workflow graph by extracting logs from the distributed system, check the graph model using temporal relationships, and convert event transition times into time vectors as weights of edges in the graph; next, fuse the subgraphs to accurately reflect the state and execution trajectory of the distributed system; finally, extract and train the information space structure, node features, and graph structure information of the workflow graph using a graph convolutional neural network to ultimately achieve anomaly detection in the workflow graph.

[0006] The specific steps of this invention to achieve the above objectives are as follows:

[0007] (1) Parse the logs generated by the distributed system, convert unstructured logs into structured logs, and extract log event templates;

[0008] (2) Based on linear temporal logic (LTL), mine the temporal relationships between events during system operation, i.e., time invariants;

[0009] (3) Construct a workflow graph for the structured distributed system logs, as follows:

[0010] (3.1) Construct a single-process finite state machine (FSM) model from the parsed structured log:

[0011] Multiple parallel tasks existing in the system log are called trajectories. Different objects in each trajectory have different processes. Suppose a system log L is generated by h processes. Construct a finite state machine model for each of these processes to obtain the i-th process P. i The corresponding organic state machine F i :

[0012] F i = <Q i E i ,δ i >,

[0013] Where i = 1, 2, ..., h, Q i F represents the finite state machine F constructed from the input logs. i The set of all states in E i Indicates process P i The set of all event instances, δ i Indicates process P i The transformation function;

[0014] (3.2) Extract the channels between processes, enable interaction between processes through the sending and receiving of information, and integrate the organic state machines corresponding to each process using the channels to generate the FSM model F:

[0015] F = <Q,E,Δ,M>

[0016] Where Q represents the set of all states in the input log's FSM, E represents the set of all event instances in log L, and Δ = {δ} i |P i ∈L} represents the set of transformation functions in log L, M={c ij |P i ,P j ∈L} represents the channel c between the i-th process and the j-th process. ij The set of i, j = 1, 2, ..., h, where i ≠ j;

[0017] (3.3) The generated FSM model is transformed to obtain the workflow diagram G:

[0018] G = (V, E)

[0019] Where, V={v i |i∈0} represents the set of nodes in G, and E={e vivj |v i ,v j∈V} represents the i-th node v in G. i Pointing to the j-th v j The set of directed edges; 0 is a positive integer;

[0020] (4) Improve the workflow diagram G according to the following steps:

[0021] (4.1) Abstract the workflow graph G, remove redundant parts, that is, treat a sequence of event instances as a whole state, merge multiple identical event instance sequences to form a new abstract state, construct a series of abstract states for all events, and obtain the relationship between states.

[0022] (4.2) Use the time invariants mined in step (2) to check the constructed workflow graph G, and modify the parts that do not conform to the temporal relationship between instances to obtain the final workflow graph model.

[0023] (5) Treat anomaly detection as a binary classification task, namely normal class or abnormal class. Use a graph convolutional neural network based on Top-K pooling mechanism as the anomaly detection model. By extracting the embedding representation of nodes in the graph and the structural information of the graph, perform graph classification task on the final workflow graph model to realize anomaly detection of distributed system logs.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] First, because this invention uses a graph convolutional neural network model to extract and train the node features, graph structure features, and spatial information features extracted from the workflow graph model, it solves the problem of ignoring the spatial structure information between log sequences in the traditional log anomaly detection process. Therefore, representing distributed system logs in the form of a workflow graph for anomaly detection can more fully consider the relationship between system execution trajectories and add it as graph spatial structure information to the graph convolutional neural network model for anomaly detection, thereby further improving the accuracy of anomaly detection in distributed systems.

[0026] Secondly, the workflow graph anomaly detection model adopted in this invention can detect various anomaly types, including performance anomalies, system sudden event anomalies, and system log execution sequence anomalies. It focuses more on the identification of system anomaly types and the location of anomaly points, solving the problem of difficulty in analyzing and detecting distributed systems, and providing a reference for the search and repair of system anomalies.

[0027] Third, this invention proposes a method for constructing and integrating workflow graphs at each end of a distributed system. First, separate workflow graph subgraphs are constructed for each computing node in the distributed system. Then, the subgraphs are merged. By mining time-series invariants in the logs, the workflow graph model is iteratively improved and refined. The final overall graph clearly and concisely reflects the execution and status of the entire distributed system, solving the problem of scattered and difficult-to-process logs in distributed systems. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the implementation of the present invention;

[0029] Figure 2 This is a schematic diagram illustrating the construction process of the workflow diagram in this invention;

[0030] Figure 3 This is a simplified example of the workflow diagram constructed in this invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the present invention will be further described clearly and completely below in conjunction with specific embodiments.

[0032] Example 1:

[0033] Reference Figure 1 This invention proposes a distributed system anomaly detection method based on log workflow mining, which specifically includes the following steps:

[0034] Step 1 involves parsing the logs generated by the distributed system, converting unstructured logs into structured logs, and extracting log event templates. This involves removing variables from log entries and retaining the text portion as the log template, thus transforming unstructured logs into structured logs for further processing. This embodiment employs an online log parsing method based on a fixed-depth tree. It preprocesses the logs using simple regular expressions based on domain knowledge, then matches log events in the log group using a fixed-depth tree to convert unstructured logs into structured logs and extract the log event templates.

[0035] Step 2: Based on Linear Temporal Logic (LTL), the temporal relationships between events during system operation are mined. This invention refers to these temporal relationships as "temporal invariants." This embodiment utilizes three LTL expressions to mine the temporal relationships existing during system operation from the distributed system logs. Specifically, it mines the constants of the temporal relationships between events during system operation. These constants are obtained from the actual system logs and therefore satisfy the real system operation scenario. When there are temporal relationships in the model that do not conform to the constants, they are removed. The temporal relationships of event instances in the logs are represented by symbols, where t1 represents the time when the first event e1 is recorded in the log, and t2 represents the time when the second event e2 is recorded. For event instance m i ∈P i ,n j ∈P j The following three constant relationships were obtained through mining:

[0036]

[0037]

[0038]

[0039] Where, m i Indicates process P i event instances, n j Indicates process P j event instances in Indicates any, Indicates existence. This indicates that it does not exist.

[0040] The timing relationship constants between events during the operation of the mining system are determined according to the following rules:

[0041] (a) If event instance m i The number of times it appears in the log and m i Appearing in n j If the number of occurrences before the event is equal, then the event instance m is considered to be... i n j Satisfy m i →n j relation.

[0042] (b) If event instance m i Appearing in n j If the previous count is 0, then the event instance m is considered to be... i n j satisfy relation.

[0043] (c) If event instance n j The number of times it appears in the log and m i Appearing in n j If the number of occurrences before the event is equal, then the event instance m is considered to be... i n j Satisfy m i ←n j relation.

[0044] Step 3: Construct a workflow graph for the structured distributed system logs, specifically implemented as follows:

[0045] First, construct an FSM for each process. For each process P i Its FSM is a tuple: F i = <Q i E i ,δ i Then, using inter-process channels, each FSM of the h processes in log L is integrated to obtain the FSM of the entire log: F=<F1,F2,...,F h Finally, it is transformed to obtain the workflow diagram.

[0046] (3.1) Construct a single-process finite state machine (FSM) model from the parsed structured log:

[0047] Multiple parallel tasks existing in the system log are called trajectories. Different objects in each trajectory have different processes. Suppose a system log L is generated by h processes. Construct a finite state machine model for each of these processes to obtain the i-th process P. i The corresponding organic state machine F i :

[0048] F i = <Q i E i ,δ i >,

[0049] Where i = 1, 2, ..., h, Q i F represents the finite state machine F constructed from the input logs. i The set of all states in E i Indicates process P i The set of all event instances, δ i Indicates process P i The transformation function;

[0050] The finite state machine F constructed from the above input logs i The set of all states Q i , means as follows:

[0051]

[0052] in, L represents the initial state of the i-th process. i For process P i In log L, s represents a sequence of event instances, where s∈L i If the event instance sequence s is a complete process record in the log, then the state after experiencing the event instance sequence s is... Let be the final state of the i-th process; t represents the sequence of event instances.

[0053] The states in the FSM are transformed according to the following formula:

[0054]

[0055] Among them, s·e i This indicates that the event instance sequence s is followed by the event instance e. i ; Indicates splicing event e i The final state of the process.

[0056] (3.2) Extract the channels between processes, enable interaction between processes through the sending and receiving of information, and integrate the organic state machines corresponding to each process using the channels to generate the FSM model F:

[0057] F = <Q,E,Δ,M>

[0058] Where Q represents the set of all states in the input log's FSM, E represents the set of all event instances in log L, and Δ = {δ} i |P i ∈L} represents the set of transformation functions in log L, M={c ij |P i ,P j ∈L} represents the channel c between the i-th process and the j-th process. ij The set of i, j = 1, 2, ..., h, and i ≠ j.

[0059] The set Q of all states in the input log's FSM is represented as follows:

[0060] Q = Q1 × Q2 × ... × Q i ×...×Q h .

[0061] For a single process, the relationships between events are completely ordered and linear. We first construct an FSM for each process, then integrate the FSMs of multiple processes to build the FSM for the entire log L. Let's assume that for process P... i and P jIntegrate with FSM, if event instance e is through process P i and P j Channel c ij Once the inter-process transfer is complete, process P... i status and process P j status The connection is completed via event e.

[0062] (3.3) The generated FSM model is transformed to obtain the workflow diagram G:

[0063] G = (V, E)

[0064] in, Describes the set of nodes in G. v represents the i-th node in G. i Pointing to the j-th v j The set of directed edges; It is a positive integer;

[0065] Step 4, improve workflow diagram G according to the following steps:

[0066] (4.1) Abstract the workflow graph G, remove redundant parts, that is, treat a sequence of event instances as a whole state, merge multiple identical event instance sequences to form a new abstract state, construct a series of abstract states for all events, and obtain the relationship between states.

[0067] (4.2) Since the abstraction of FSM is highly generalized, it may lose the detailed condition features of a single-process FSM. Therefore, the FSM constructed in step 3 may have a trajectory that satisfies the state transition rules of the inference model but violates the actual system execution status. Therefore, the time invariants mined in step (2) are used to check the constructed workflow graph G, and the parts that do not conform to the temporal relationship between instances are modified and refined to obtain the final workflow graph model.

[0068] Step 5: Treat anomaly detection as a binary classification task, i.e., normal class or abnormal class. Use a graph convolutional neural network based on Top-K pooling mechanism as the anomaly detection model. By extracting the embedding representations of nodes in the graph and the structural information of the graph, perform a graph classification task on the final workflow graph model to achieve anomaly detection in the distributed system logs. The specific steps for performing the graph classification task on the final workflow graph model are as follows:

[0069] (5.1) The self-attention value Z of the graph convolutional neural network based on the Top-K pooling mechanism is calculated according to the following formula:

[0070]

[0071] in σ is the activation function. Let I be the adjacency matrix of the graph with self-loops. N It is the identity matrix. It is a degree matrix, where X is the F-dimensional input feature of N nodes in the graph, and Θ is... att Indicates the parameters of the self-attention graph pooling layer;

[0072] (5.2) Select nodes based on the self-attention value Z, as shown in the following formula:

[0073]

[0074] Where k∈(0,1] represents the pooling ratio. This represents the node selected from the self-attention values, and top-rank indicates the node that is among the nodes returned in Z. The index of the preceding node, Z mask Indicates a feature attention mask. idx Indicates index operation;

[0075] (5.3) Perform pooling on the graph using the following formula:

[0076] X' = ​​X idx,: ,X out =X'Z mask A out =A idx,idx

[0077] Among them, X idx,: Represents the feature matrix indexed by row, and represents the element-wise multiplication for the broadcast operation, which is used to calculate the similarity score between nodes in order to select the nodes to retain during graph pooling; A idx,idx This represents an adjacency matrix expressed using row and column indices, where the row and column indices represent the row and column numbers of the nodes, respectively; X out Represents the characteristic matrix, A out Represents the adjacency matrix;

[0078] (5.4) The readout layer forms a fixed-size representation by aggregating node features, as shown in the following formula:

[0079]

[0080] Where N represents the number of nodes, x i Let || represent the feature vector of the i-th node, and || represent the concatenation of vectors. The readout layer summarizes and represents the node features in the graph structure as a whole, and obtains the feature representation s of the entire graph.

[0081] (5.5) Classify the workflow graph. After obtaining the output s of the readout layer, use it as the input to the graph classification task. Map s through a fully connected layer and use the softmax function to predict the class probability distribution of the graph, as shown in the following formula:

[0082] y = softmax(Ws + b)

[0083] Where y represents the probability vector, W represents the weight matrix of the fully connected layer, and b represents the bias vector;

[0084] (5.6) The parameters in the graph convolutional neural network model are optimized using the cross-entropy loss function, which is expressed as follows:

[0085]

[0086] Among them, y i y represents the true label of graph i. i The label represents the model's predicted graph i; starting from the output layer, the parameter W is calculated in each layer. l The gradient of the loss function is used to optimize the parameters in each iteration update using the gradient descent algorithm.

[0087] (5.7) The labels of the workflow diagram are divided into normal or abnormal sets by the final output of the softmax function. The abnormality of the workflow diagram is determined according to the set in which the workflow diagram labels are located, thus realizing the system abnormality detection.

[0088] Example 2: The overall implementation steps of this example are the same as in Example 1, please refer to the appendix. Figure 2 In this embodiment, the process of constructing the workflow graph model based on structured logs is further described:

[0089] A system log contains logs generated by multiple processes. These processes, which need to be completed in parallel, constitute the system's trajectory. Finite state machines are constructed for each of these linear processes, with process P... i This represents a sequence of events occurring on the same object during the execution of a specific task. Each process execution produces a sequence of event instances, which can be the currently executing event, the current state, or transmitted information. Trajectory T i This indicates that multiple objects collaborate to complete a series of tasks, forming a set of multiple processes. A single log entry can record multiple tracks.

[0090] For a single process, the events are in a completely ordered linear relationship. First, consider each process P... i Construct FSMs separately, and then integrate the multi-process FSMs to build the FSM for log L.

[0091] This invention defines process state in two ways: local process state and global process state. Indicates the process from its initial state The state is entered after experiencing a series of event instance sequences s.

[0092] Let L i For process P i The records in log L represent the set of local process states Q. i The formula is defined as follows:

[0093]

[0094] Where, when s∈L i If the event instance sequence s is a complete process record in the log, then the state after experiencing event sequence s is... This represents the final state of the process.

[0095] The global process state is a global description of the state of all processes in log L. That is, in a log containing h processes, the global state of these h processes at a certain moment is represented by an h-element tuple q, where q=<q1,q2,...,q h >. Therefore, the formula for the set Q of all global process states is as follows:

[0096] q∈Q=Q1×Q2×...×Q h ,q i ∈Q i

[0097] After defining the states in the FSM, the next step is the transition between states. The state transition formula is defined as follows:

[0098]

[0099] Among them, s·e i This indicates that the event instance sequence s is followed by the event instance e. i ; Indicates splicing event e i The final state of the process.

[0100] Example 3: The overall implementation steps of this example are the same as those of Example 1, please refer to the appendix. Figure 3 In this embodiment, the process of improving the workflow diagram generated in step 3 is further described as follows:

[0101] Since the workflow diagram generated in step 3 contains all states and events, it is necessary to abstract and simplify the workflow diagram model in order to understand and apply it more efficiently.

[0102] The generated workflow model is abstracted to remove redundancy, making the model easier to understand and apply. In the previous model, all linear relationships were extracted and connected through states and transition functions. However, when abstracting the FSM, a sequence of event instances is treated as a whole state, and relationships between these new states are constructed. The abstraction method of this invention employs the strategy of manually setting a parameter k. If the first k states (including transition paths, transmitted information, etc.) of several sets of event instance sequences are the same, then these sequences are considered to be merged into an abstract part, forming a new abstract state.

[0103] To address the issue that the abstraction of the workflow diagram in the previous step led to a model that deviated from the actual system execution, it is necessary to mine constants from the logs that meet the actual operating conditions in order to verify and correct the model.

[0104] This embodiment uses the model checking tool McScM to verify the FSM model obtained in the first step. It employs a negative example-guided abstraction and refinement method, iteratively checking whether the model satisfies all time invariants mined in the previous step. If a non-satisfied state transition relationship is found, this negative example is removed until the model satisfies all time invariants. The corrected and refined model satisfies all time invariants mined from the logs, and the workflow graph model at this point can concisely and accurately represent the system's execution.

[0105] Example 4: The overall implementation steps of this example are the same as in Example 1. The implementation process of step 5 is further described in this example:

[0106] The aforementioned method is used to construct workflow graphs for distributed systems in batches. These workflow graphs visualize the system's operation process, and graph convolutional neural networks (GNNs) are used for pre-training on the data. GNNs possess non-translation invariance, enabling them to process non-Euclidean spatial data such as graphs, and they perform excellently on current whole-graph classification tasks. The constructed workflow graph data is input into the GNN model, employing a TopK pooling-based model, which is an improvement on traditional GNNs. In this application scenario, node importance is evaluated by combining node feature representations, and unimportant nodes are filtered out to improve the robustness of anomaly detection.

[0107] Attention mechanisms have been widely applied in various models in deep learning. The method for calculating self-attention values ​​using graph convolution in this invention is as follows:

[0108]

[0109] in σ is the activation function; common activation functions include ReLU and tanh. Let I be the adjacency matrix of the graph with self-loops. N It is the identity matrix. It is a degree matrix, where X is the F-dimensional input feature of N nodes in the graph, and Θ is... att This represents the parameters of the self-attention graph pooling layer.

[0110] The self-attention value obtained by using graph convolution incorporates the node features and topological structure of the graph. Next, nodes are selected using the following formula:

[0111]

[0112] Pooling rate k∈(0,1], then the self-attention value Z obtained by the convolution of the previous graph is the first... The nodes will be selected. The purpose of top-rank is to return the nodes in Z that are ranked in the top rank. The index of the preceding node, the feature attention mask is defined as Z. mask .

[0113] The formula for graph pooling is as follows:

[0114] X' = ​​X idx ,X out =X'⊙Z mask A out =A idx,idx

[0115] Among them, X idx It is the feature matrix indexed by node, ⊙ is the positional inner product, A idx,idx It is an adjacency matrix indexed by row and column. X out A out These are the new feature matrix and the corresponding adjacency matrix, respectively.

[0116] The Readout layer aggregates node features to form a fixed-size representation, as shown in the following formula:

[0117]

[0118] Where N represents the number of nodes, x i Let represent the feature vector of the i-th node, and || denote the concatenation of vectors. The readout layer summarizes and represents the node features in the graph structure as a whole, obtaining the feature representation s of the entire graph. Since this paper performs overall graph labeling, normal graphs are labeled as 0, and abnormal graphs are labeled as 1. The training process consists of two modules: forward propagation and backward propagation.

[0119] ① Forward propagation: Node features and edge features are passed sequentially from the input layer to each layer until the output is obtained. The output is obtained through convolutional layers, pooling layers, and fully connected layers, and non-linearization is achieved using activation functions. Finally, the softmax function is used to output the category of the graph.

[0120] ② Backpropagation: Starting from the output layer, calculate the parameter W in each layer. l The gradient of the loss function is used to optimize the parameters in each iteration of the gradient descent algorithm.

[0121] The output of the final softmax function can be used to classify the labels of the workflow graph into normal or abnormal sets, thereby enabling anomaly detection in the workflow graph.

[0122] The parts of this invention not described in detail are common knowledge to those skilled in the art.

[0123] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Obviously, those skilled in the art, after understanding the content and principle of the present invention, may make various modifications and changes in form and detail without departing from the principle and structure of the present invention. However, these modifications and changes based on the concept of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A distributed system anomaly detection method based on log workflow mining, characterized in that, Includes the following steps: (1) Parse the logs generated by the distributed system, convert unstructured logs into structured logs, and extract log event templates; (2) Based on linear temporal logic (LTL), mine the temporal relationships between events during system operation, i.e., time invariants; (3) Construct a workflow diagram for the structured distributed system logs, as follows: (3.1) Construct a single-process finite state machine (FSM) model from the parsed structured log: Multiple parallel tasks existing in the system log are called trajectories. Different objects in each trajectory have different processes. Let's say a system log... Depend on The nth process is generated, and finite state machine models are constructed for each of these processes to obtain the nth process. Processes Corresponding finite state machine : , in, , Represents a finite state machine constructed from input logs. The set of all states in the set. Represents a process A collection of all event instances. Represents a process The transformation function; (3.2) Extract the channels between processes, enable interaction between processes through the sending and receiving of information, and integrate the finite state machines corresponding to each process using the channels to generate the FSM model. : , in, This represents the set of all states in the FSM of the input log. Represents log A collection of all event instances. Represents log The set of transformation functions in Chinese. Indicates the first The process and the first Inter-process channels The set, ; (3.3) Convert the generated FSM model to obtain the workflow diagram. : , in, express The set of nodes, express The Middle Nodes Pointing to the indivual The set of directed edges; It is a positive integer; (4) Workflow diagram Make improvements following these steps: (4.1) Workflow diagram Abstraction is performed to remove redundant parts. That is, a sequence of event instances is treated as a whole state. Multiple sequences of the same event instances are merged to form a new abstract state. A series of abstract states are constructed for all events, and the relationships between states are obtained. (4.2) Construct a workflow diagram using the time invariants extracted in step (2). The process involves checking and modifying any parts that do not conform to the temporal relationships between instances to obtain the final workflow diagram model. (5) Treating anomaly detection as a binary classification task, i.e., normal class or abnormal class, a graph convolutional neural network based on Top-K pooling mechanism is used as the anomaly detection model. By extracting the embedding representation of nodes in the graph and the structural information of the graph, a graph classification task is performed on the final workflow graph model to realize anomaly detection of distributed system logs; the graph classification task performed on the final workflow graph model is as follows: (5.1) The self-attention value of the graph convolutional neural network based on the Top-K pooling mechanism is calculated according to the following formula. : , in , , For activation function, Let be the adjacency matrix of the graph with self-loops. It is the identity matrix. It is a degree matrix. For the diagram Nodes 3D input features, Indicates the parameters of the self-attention graph pooling layer; (5.2) Based on self-attention value The formula for selecting nodes is as follows: , ; in, Indicates pooling rate. This indicates the node selected from the attention values. Indicates return In China The index of the previous node, Indicates a feature attention mask. Indicates index operation; (5.3) Perform pooling on the graph using the following formula: , ; in, This represents the feature matrix indexed by row. This indicates element-wise multiplication for broadcast operations; This represents an adjacency matrix represented by row and column indices, where the row index and column index represent the row number and column number of the node, respectively. Representing the characteristic matrix, Represents the adjacency matrix; (5.4) The readout layer forms a fixed-size representation by aggregating node features, as shown in the following formula: ; in, Indicates the number of nodes. Indicates the first The feature vector of each node The readout layer represents the connection of vectors and summarizes and represents the node features in the graph structure as a whole, thus obtaining the feature representation of the entire graph. ; (5.5) Classify the workflow diagram and obtain the output of the readout layer. Then, it is used as input for a graph classification task. Mapping is performed through a fully connected layer, utilizing... The class probability distribution of the function prediction graph is given by the following formula: ; in, Represents a probability vector. This represents the weight matrix of the fully connected layer. Represents the bias vector; (5.6) The parameters in the graph convolutional neural network model are optimized using the cross-entropy loss function, which is expressed as follows: ; in, Representation diagram The true label, Represents the model prediction graph The labels; starting from the output layer, calculate the parameters in each layer. The gradient of the loss function is used to optimize the parameters in each iteration update using the gradient descent algorithm. (5.7) Through the final The function output categorizes the workflow diagram labels into normal or abnormal sets. Based on the set where the workflow diagram labels are located, it determines whether the workflow diagram is abnormal, thus realizing system anomaly detection.

2. The method according to claim 1, characterized in that: Step (2) Based on the linear temporal logic (LTL) mining, the temporal relationship between events during the system operation is mined. The constant is mined from the actual system log. When there is a temporal relationship in the model that does not conform to the constant, it is removed. Use symbols to represent the temporal relationships of event instances in the log. To indicate, to make Indicates the first event The time recorded in the log Represents the second event The recorded time , For event instances The following three constant relationships were obtained through mining: ; ; ; in, Represents a process event instances in Represents a process event instances in Indicates any, Indicates existence. This indicates that it does not exist.

3. The method according to claim 2, characterized in that: The timing relationship constants between events during the operation of the mining system are determined according to the following rules: (a) If the event instance The number of times it appears in the log and Appeared If the number of occurrences is equal, then the event instance is considered to be an event instance. , satisfy relation; (b) If the event instance Appeared If the previous count is 0, then the event instance is considered to be... , satisfy relation; (c) If the event instance The number of times it appears in the log and Appeared If the number of occurrences is equal, then the event instance is considered to be an event instance. , satisfy relation.

4. The method according to claim 1, characterized in that: The finite state machine constructed from the input log in step (3.1) The set of all states in , means as follows: ; in, Indicates the first The initial state of each process. For process In the log The records in Represents a sequence of event instances, when That is, the sequence of event instances For a complete process record in the log, it goes through a sequence of event instances. The state after For the first The final state of each process; Represents a sequence of event instances.

5. The method according to claim 4, characterized in that: The states in the FSM are transformed according to the following formula: ; in, Represents a sequence of event instances Post-concatenation event instance ; Indicates splicing event The final state of the process.

6. The method according to claim 1, characterized in that: In step (3.2), the input log contains the set of all states in the FSM. , means as follows: 。

Citation Information

Patent Citations

  • Anomaly detection method and device for distributed unmanned aerial vehicle group weblog analysis

    CN112860533A

  • Log anomaly detection method and system based on knowledge base and ensemble learning

    CN115062144A