Log anomaly detection method and device, equipment, storage medium and program product

Abstract

The invention relates to the technical field of computers, and provides a log anomaly detection method and device, equipment, a storage medium and a program product. The method comprises the steps of constructing a time sequence event graph based on a log event sequence; determining an adjacent matrix of each event graph snapshot and a node feature matrix of each event graph snapshot; inputting each event graph snapshot, each adjacent matrix and each node feature matrix into a dynamic graph model, and performing time sequence feature extraction and link prediction to obtain a predicted event graph snapshot; and judging whether the log event sequence is abnormal or not based on the predicted event graph snapshot. By means of the mode, efficient detection and positioning of the abnormal log can be achieved, the detection effect of the abnormal log is improved, and the detection requirement of an information system for log abnormity is met.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a log anomaly detection method, device, equipment, storage medium and program product. Background Art

[0002] With the development of information technology, the application scope of information equipment and systems is becoming wider and wider, which has generated a large amount of log information. These log information records the behavior and status of system operation, providing an important basis for judging system anomalies. However, it is a very challenging task to mine errors and dangerous operation behaviors from unstructured system log information and analyze whether log events are abnormal.

[0003] In the prior art, log anomaly detection methods are mainly divided into detection methods based on log event counting and detection methods based on log event temporal dependency.

[0004] The detection method based on log event count is to convert the parsed log event sequence into the form of event count vectors, and use traditional machine learning algorithms such as logistic regression to identify abnormal behavior patterns in the log event sequence. However, this method cannot build the temporal dependency between multi-source logs and cannot capture long-term abnormal operation behaviors in the log event sequence.

[0005] The detection method based on the time-series dependency of log events is to design the corresponding deep model according to the order of occurrence of each log event in the log event sequence, and use time-series learning models such as recursive neural networks to capture the time-series evolution relationship between log events, and learn the abnormal operation behavior pattern based on time-series features. However, this method often cannot capture the evolution relationship between each log event, and the model cannot perceive the correlation relationship of the latest log events, which leads to the inability to guarantee the detection performance of the model.

[0006] Therefore, the existing log anomaly detection methods have poor detection effects and are difficult to meet the information system's needs for log anomaly detection. Summary of the invention

[0007] The embodiments of the present application provide a log anomaly detection method, apparatus, device, storage medium and program product to solve the technical problem that the existing log anomaly detection method has poor detection effect and is difficult to meet the information system's detection requirements for log anomalies.

[0008] In a first aspect, an embodiment of the present application provides a log anomaly detection method, comprising: constructing a time series event graph based on a log event sequence; the time series event graph includes multiple event graph snapshots; determining an adjacency matrix of each event graph snapshot and a node feature matrix of each event graph snapshot; the adjacency matrix and the node feature matrix are used to characterize the spatial features of the event graph snapshot; inputting each event graph snapshot, each adjacency matrix and each node feature matrix into a pre-trained dynamic graph model, performing time series feature extraction and link prediction, and obtaining a predicted event graph snapshot output by the dynamic graph model; the predicted event graph snapshot is an event graph snapshot at a future moment, and the predicted event graph snapshot is used to characterize potential new relationships between log events; based on the predicted event graph snapshot, determining whether the log event sequence is abnormal.

[0009] In one embodiment, a log event sequence includes multiple event sequences arranged in sequence according to timestamps; based on the log event sequence, a time series event graph is constructed, including: based on the event relationship of each event sequence, an event graph snapshot of each event sequence is constructed in sequence; an event graph snapshot includes a node set and an edge flow set, the node set includes multiple nodes, the edge flow set includes multiple edges, the nodes are used to represent log events, and the edges are used to represent the execution order of log events; based on each event graph snapshot, a time series event graph is constructed; the multiple event graph snapshots in the time series event graph are arranged in sequence according to timestamps.

[0010] In one embodiment, the adjacency matrix of each event graph snapshot and the node feature matrix of each event graph snapshot are determined, including: inputting each event graph snapshot into a pre-trained edge flow model, obtaining the adjacency matrix of each event graph snapshot and the node feature matrix of each event graph snapshot output by the edge flow model; wherein the edge flow model is used to update node features according to the edge flow set of adjacent event graph snapshots, and the node feature matrix corresponding to an event graph snapshot is obtained based on the corresponding edge flow set and the initial node feature update, and the initial node feature is the node feature matrix corresponding to the previous event graph snapshot.

[0011] In one embodiment, the dynamic graph model includes a graph neural network, a Transformer network and a fully connected network; the graph neural network is used to extract time series features based on each event graph snapshot, each adjacency matrix and each node feature matrix to generate a first feature of each event graph snapshot; the Transformer network is used to add position encoding to the time series features of each event graph snapshot to generate a second feature of each event graph snapshot; log event feature learning and splicing are performed on the second feature of each event graph snapshot to generate a third feature; the fully connected network is used to perform link prediction based on the third feature to generate a predicted event graph snapshot.

[0012] In one embodiment, based on the predicted event graph snapshot, determining whether a log event sequence is abnormal includes: determining a scoring rule based on the predicted event graph snapshot, and scoring the log event sequence based on the scoring rule to obtain a log sequence score; determining whether the log sequence score is greater than or equal to an abnormal score threshold; if the log sequence score is greater than or equal to the abnormal score threshold, determining that the log event sequence is abnormal.

[0013] In one embodiment, based on the predicted event graph snapshot, determining whether an abnormality occurs in the log event sequence includes: determining a scoring rule based on the predicted event graph snapshot, and scoring the log event sequence based on the scoring rule to obtain a log sequence score; determining whether the log sequence score is greater than or equal to an abnormality score threshold; if the log sequence score is less than the abnormality score threshold, determining that no abnormality occurs in the log event sequence.

[0014] In the second aspect, an embodiment of the present application provides a log anomaly detection device, including: a time series event graph construction module, which is used to construct a time series event graph based on a log event sequence; the time series event graph includes multiple event graph snapshots; a feature update module, which is used to determine the adjacency matrix of each event graph snapshot and the node feature matrix of each event graph snapshot; the adjacency matrix and the node feature matrix are used to characterize the spatial features of the event graph snapshot; a link prediction module, which is used to input each event graph snapshot, each adjacency matrix and each node feature matrix into a pre-trained dynamic graph model, perform time series feature extraction and link prediction, and obtain a predicted event graph snapshot output by the dynamic graph model; the predicted event graph snapshot is an event graph snapshot at a future moment, and the predicted event graph snapshot is used to characterize potential new relationships between log events; an anomaly detection module, which is used to determine whether an anomaly occurs in the log event sequence based on the predicted event graph snapshot.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned log anomaly detection methods is implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned log anomaly detection methods.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any of the above-mentioned log anomaly detection methods when executed by a processor.

[0018] The log anomaly detection method, apparatus, device, storage medium and program product provided in the embodiments of the present application construct a time series event graph based on a log event sequence; the time series event graph includes multiple event graph snapshots; the adjacency matrix of each event graph snapshot and the node feature matrix of each event graph snapshot are determined; the adjacency matrix and the node feature matrix are used to characterize the spatial features of the event graph snapshot; each event graph snapshot, each adjacency matrix and each node feature matrix are input into a pre-trained dynamic graph model, and time series feature extraction and link prediction are performed to obtain a predicted event graph snapshot output by the dynamic graph model; the predicted event graph snapshot is an event graph snapshot at a future moment, and the predicted event graph snapshot is used to characterize the potential new relationship between log events; based on the predicted event graph snapshot, it is determined whether the log event sequence is abnormal. Through the above method, in the process of log anomaly detection, the influence of information in two dimensions, spatial features and temporal features, on log detection is comprehensively considered, which can effectively improve the model's analysis depth of log event sequences and obtain more comprehensive log event features. As a result, the model can mine the attack modes of abnormal events in log event sequences from two dimensions, spatial features and temporal features, capture potential new relationships between log events, generate predicted event graph snapshots, and realize efficient detection and positioning of abnormal logs based on predicted event graph snapshots, thereby improving the detection effect of abnormal logs and meeting the information system's detection needs for log anomalies. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 This is one of the flow charts of the log anomaly detection method provided in the embodiment of the present application.

[0021] Figure 2 This is the second flow chart of the log anomaly detection method provided in the embodiment of the present application.

[0022] Figure 3 This is the third flow chart of the log anomaly detection method provided in the embodiment of the present application.

[0023] Figure 4 It is a schematic diagram of the construction process of the timing event diagram provided in the embodiment of the present application.

[0024] Figure 5 It is a schematic diagram of the log event sequence scoring process provided in an embodiment of the present application.

[0025] Figure 6It is a structural diagram of a log anomaly detection device provided in an embodiment of the present application.

[0026] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] See also Figures 1 to 3 , Figure 1 This is one of the flow charts of the log anomaly detection method provided in the embodiment of the present application. Figure 2 This is the second flow chart of the log anomaly detection method provided in the embodiment of the present application. Figure 3 This is the third flow chart of the log anomaly detection method provided in the embodiment of the present application. Figure 1 As shown, in an embodiment of the present application, a log anomaly detection method is applied to an information system or device, and the log anomaly detection method includes steps S110 to S140, and each step is specifically as follows: S110: Construct a time series event graph based on the log event sequence.

[0029] A time series event graph includes multiple event graph snapshots.

[0030] Among them, the temporal event graph is constructed based on the log event sequence that describes the temporal development of system events in the form of logs. Each node in the temporal event graph represents a log event, and the edge represents the sequential relationship between different log events (that is, the execution order of log events).

[0031] Specifically, Figure 2 and Figure 3 As shown, the log event sequence It includes multiple event sequences arranged in sequence according to timestamps, denoted as ,in, Indicates A sequence of timestamped events, Indicates The first event in the sequence In order to capture the complex relationship between log events, it is necessary to convert the event sequence into a time series event graph. ,in, express A snapshot of the event graph at time instant, Represents a collection of nodes. Represents an edge flow set (i.e., edge set).

[0032] Furthermore, based on the event relationship of each event sequence, an event graph snapshot of each event sequence may be constructed in sequence, and based on each event graph snapshot, a time series event graph may be constructed.

[0033] Among them, an event graph snapshot includes a node set and an edge flow set. The node set includes multiple nodes, and the edge flow set includes multiple edges. The nodes are used to represent log events, and the edges are used to represent the execution order of log events. Multiple event graph snapshots in the time series event graph are arranged in sequence according to timestamps.

[0034] See also Figure 4 , Figure 4 It is a schematic diagram of the construction process of the timing event diagram provided in the embodiment of the present application.

[0035] like Figure 4 As shown, assuming that a single event sequence is used as a unit to construct a first-order sequential event diagram, we can first construct a sequential event diagram based on the first event sequence. Build the first event graph snapshot by using the event relationships in ; Then, the second event sequence New event relationships are added to the previous event graph snapshot In the second event graph snapshot ; and so on, until an event graph snapshot containing all event relationships is generated , thus constructing a complete time series event graph Similarly, the construction of the second-order temporal event diagram is based on the second-order event relationship.

[0036] It should be noted that, while constructing the time series event graph, the embodiment of the present application records the dynamic edge flow information contained in each event graph snapshot. Can be Dynamic edge flow of time intervals Composition, among which, yes The edge that appears at any moment, represents the source node, Represents the target node.

[0037] It can be understood that in actual application, the order of the time series event diagram can be adjusted according to actual needs.

[0038] like Figure 4 As shown, the first event graph snapshot of the first-order sequential event graph is constructed. When the first event sequence The six event relationships that appear in the event graph are added to the event graph snapshot At the same time, these six event relations are expressed as edge flows, which are recorded as ( )、( )、( ) 、( )、( )、( ).

[0039] Similarly, when constructing other event graph snapshots, while adding the newly appeared event relations to the previous event graph snapshot, all the appearing event relations should also be represented as dynamic edge flows.

[0040] S120: Determine an adjacency matrix of each event graph snapshot and a node feature matrix of each event graph snapshot.

[0041] The adjacency matrix and node feature matrix are used to characterize the spatial characteristics of the event graph snapshot.

[0042] like Figure 2 and Figure 3 As shown in the timing event diagram After that, the node features need to be dynamically updated based on the edge flow information. The event graph snapshots at different times in the time series event graph have a fixed graph structure, and the adjacency matrix corresponding to each event graph snapshot can be recorded as ,in, Indicates The adjacency matrix of the event graph snapshot is used to represent the events that contain In order to further capture the features of each event graph snapshot, it is also necessary to obtain the node feature matrix of each event graph snapshot .

[0043] In this embodiment, the node feature matrix of each event graph snapshot is updated by the edge flow model. The edge flow model can use the edge flow information (ie, edge flow set) of the event graph snapshot to update the node features in real time, thereby improving the accuracy of the model.

[0044] Among them, the edge flow model can update node features according to the edge flow set of adjacent event graph snapshots. The node feature matrix corresponding to an event graph snapshot is obtained based on the corresponding edge flow set and the initial node feature update. The initial node feature is the node feature matrix corresponding to the previous event graph snapshot.

[0045] For example, in the event graph snapshot In this event graph snapshot Initial node features The previous event graph snapshot The final updated node feature matrix To get a snapshot of the event graph The final node feature needs to be processed for each edge in its corresponding edge flow set to update the event graph snapshot The final node feature matrix First, when the event graph snapshot The corresponding edge flow set The edge When it appears, according to Central source node Node features and the target node Node features Perform feature aggregation to satisfy the following formula: ; in, Indicates the target node In the Event graph snapshot The initial eigenvector in ; Represents the source node In the Event graph snapshot The initial eigenvector in ; Indicates that after the edge flow is updated, the target node The new eigenvector of ; represents the activation function; and are different weight coefficients. After it appears, by Perform reverse time walk to the target node Performing forward time walks can update node features, which means that in log events After that, the log event , which is similar to the log event Directly related to the log event that occurred at the previous moment.

[0046] Optionally, in order to better train the edge flow model, it is also necessary to select a reasonable loss function for pre-training. Combining the design method of the loss function in the dynamic graph neural network model and considering the practical significance of the time series event graph, this embodiment designs the following loss function: ; ; ; in, Indicates one side The loss function value of Representation Node In the previous event graph snapshot The eigenvectors in ; Representation Node Snapshot of the current event graph The eigenvectors in ; Represents the current event graph snapshot The total loss is the loss of all edges The sum is obtained; Represents edge The set of nodes involved in all feature updates after the appearance; Representation Node The inner product of the node features before feature update and the node features after feature update; is a snapshot of the event graph The edge flow set.

[0047] S130: Input each event graph snapshot, each adjacency matrix and each node feature matrix into the pre-trained dynamic graph model, perform time series feature extraction and link prediction, and obtain a predicted event graph snapshot output by the dynamic graph model.

[0048] The predicted event graph snapshot is a snapshot of the event graph at a future moment, and is used to characterize potential new relationships between log events.

[0049] S140: Based on the predicted event graph snapshot, determine whether the log event sequence is abnormal.

[0050] The log anomaly detection method provided in the embodiment of the present application constructs a time series event graph based on a log event sequence; the time series event graph includes multiple event graph snapshots; the adjacency matrix of each event graph snapshot and the node feature matrix of each event graph snapshot are determined; the adjacency matrix and the node feature matrix are used to characterize the spatial features of the event graph snapshot; each event graph snapshot, each adjacency matrix and each node feature matrix are input into a pre-trained dynamic graph model, and time series feature extraction and link prediction are performed to obtain a predicted event graph snapshot output by the dynamic graph model; the predicted event graph snapshot is an event graph snapshot at a future moment, and the predicted event graph snapshot is used to characterize potential new relationships between log events; based on the predicted event graph snapshot, it is determined whether the log event sequence is abnormal. Through the above method, in the process of log anomaly detection, the influence of information in two dimensions, spatial features and temporal features, on log detection is comprehensively considered, which can effectively improve the model's analysis depth of log event sequences and obtain more comprehensive log event features. As a result, the model can mine the attack modes of abnormal events in log event sequences from two dimensions, spatial features and temporal features, capture potential new relationships between log events, generate predicted event graph snapshots, and realize efficient detection and positioning of abnormal logs based on predicted event graph snapshots, thereby improving the detection effect of abnormal logs and meeting the information system's detection needs for log anomalies.

[0051] In some embodiments, a log event sequence includes multiple event sequences arranged in sequence according to timestamps; based on the log event sequence, a time series event graph is constructed, including: based on the event relationship of each event sequence, an event graph snapshot of each event sequence is constructed in sequence; an event graph snapshot includes a node set and an edge flow set, the node set includes multiple nodes, the edge flow set includes multiple edges, the nodes are used to represent log events, and the edges are used to represent the execution order of log events; based on each event graph snapshot, a time series event graph is constructed; the multiple event graph snapshots in the time series event graph are arranged in sequence according to timestamps.

[0052] The temporal event graph is constructed based on the log event sequence that describes the temporal development of system events in the form of logs. Each node in the temporal event graph represents a log event, and the edge represents the sequential relationship between different log events (that is, the execution order of log events).

[0053] Specifically, Figure 2 and Figure 3 As shown, the log event sequence It includes multiple event sequences arranged in sequence according to timestamps, denoted as ,in, Indicates A sequence of timestamped events, Indicates The first event in the sequence In order to capture the complex relationship between log events, it is necessary to convert the event sequence into a time series event graph. ,in, express A snapshot of the event graph at time instant, Represents a collection of nodes. Represents an edge flow set (i.e., edge set).

[0054] Furthermore, based on the event relationship of each event sequence, an event graph snapshot of each event sequence may be constructed in sequence, and based on each event graph snapshot, a time series event graph may be constructed.

[0055] Among them, an event graph snapshot includes a node set and an edge flow set. The node set includes multiple nodes, and the edge flow set includes multiple edges. The nodes are used to represent log events, and the edges are used to represent the execution order of log events. Multiple event graph snapshots in the time series event graph are arranged in sequence according to timestamps.

[0056] Specifically, assuming that a single event sequence is used as a unit to construct a first-order temporal event graph, we can first construct the first event sequence based on the Build the first event graph snapshot by using the event relationships in ; Then, the second event sequence New event relationships are added to the previous event graph snapshot In the second event graph snapshot ; and so on, until an event graph snapshot containing all event relationships is generated , thus constructing a complete time series event graph Similarly, the construction of the second-order temporal event diagram is based on the second-order event relationship.

[0057] It should be noted that, while constructing the time series event graph, the embodiment of the present application records the dynamic edge flow information contained in each event graph snapshot. Can be Dynamic edge flow of time intervals Composition, among which, yes The edge that appears at any moment, represents the source node, Represents the target node.

[0058] It can be understood that in actual application, the order of the time series event diagram can be adjusted according to actual needs.

[0059] Similarly, when constructing other event graph snapshots, while adding the newly appeared event relations to the previous event graph snapshot, all the appearing event relations should also be represented as dynamic edge flows.

[0060] In some embodiments, the adjacency matrix of each event graph snapshot and the node feature matrix of each event graph snapshot are determined, including: inputting each event graph snapshot into a pre-trained edge flow model, obtaining the adjacency matrix of each event graph snapshot and the node feature matrix of each event graph snapshot output by the edge flow model; wherein the edge flow model is used to update node features according to the edge flow set of adjacent event graph snapshots, and the node feature matrix corresponding to an event graph snapshot is obtained based on the corresponding edge flow set and the initial node feature update, and the initial node feature is the node feature matrix corresponding to the previous event graph snapshot.

[0061] like Figure 2 and Figure 3 As shown in the timing event diagram After that, the node features need to be dynamically updated based on the edge flow information. The event graph snapshots at different times in the time series event graph have a fixed graph structure, and the adjacency matrix corresponding to each event graph snapshot can be recorded as ,in, Indicates The adjacency matrix of the event graph snapshot is used to represent the events that contain In order to further capture the features of each event graph snapshot, it is also necessary to obtain the node feature matrix of each event graph snapshot .

[0062] In this embodiment, the node feature matrix of each event graph snapshot is updated by the edge flow model. The edge flow model can use the edge flow information (ie, edge flow set) of the event graph snapshot to update the node features in real time, thereby improving the accuracy of the model.

[0063] Among them, the edge flow model can update node features according to the edge flow set of adjacent event graph snapshots. The node feature matrix corresponding to an event graph snapshot is obtained based on the corresponding edge flow set and the initial node feature update. The initial node feature is the node feature matrix corresponding to the previous event graph snapshot.

[0064] For example, in the event graph snapshot In this event graph snapshot Initial node features The previous event graph snapshot The final updated node feature matrix To get a snapshot of the event graph The final node feature needs to be processed for each edge in its corresponding edge flow set to update the event graph snapshot The final node feature matrix First, when the event graph snapshot The corresponding edge flow set The edge When it appears, according to Central source node Node features and the target node Node features Perform feature aggregation to satisfy the following formula: ; in, Indicates the target node In the Event graph snapshot The initial eigenvector in ; Represents the source node In the Event graph snapshot The initial eigenvector in ; Indicates that after the edge flow is updated, the target node The new eigenvector of ; represents the activation function; and are different weight coefficients. After it appears, by Perform reverse time walk to the target node Performing forward time walks can update node features, which means that in log events After that, the log event , which is similar to the log event Directly related to the log event that occurred at the previous moment.

[0065] Optionally, in order to better train the edge flow model, it is also necessary to select a reasonable loss function for pre-training. Combining the design method of the loss function in the dynamic graph neural network model and considering the practical significance of the time series event graph, this embodiment designs the following loss function: ; ; ; in, Indicates one side The loss function value of Representation Node In the previous event graph snapshot The eigenvectors in ; Representation Node Snapshot of the current event graph The eigenvectors in ; Represents the current event graph snapshot The total loss is the loss of all edges The sum is obtained; Represents edge The set of nodes involved in all feature updates after the appearance; Representation Node The inner product of the node features before feature update and the node features after feature update; is a snapshot of the event graph The edge flow set.

[0066] In some embodiments, the dynamic graph model includes a graph neural network, a Transformer network and a fully connected network; the graph neural network is used to extract time series features based on each event graph snapshot, each adjacency matrix and each node feature matrix to generate a first feature of each event graph snapshot; the Transformer network is used to add position encoding to the time series features of each event graph snapshot to generate a second feature of each event graph snapshot; log event feature learning and splicing are performed on the second feature of each event graph snapshot to generate a third feature; the fully connected network is used to perform link prediction based on the third feature to generate a predicted event graph snapshot.

[0067] like Figure 2 and Figure 3 As shown in the timing event diagram , and the adjacency matrix corresponding to each event graph snapshot and the node feature matrix of each event graph snapshot Afterwards, it is necessary to perform link prediction based on a dynamic graph neural network to capture the topological features of a single event graph snapshot in the time series event graph and their dynamic evolution process, and generate the predicted event graph at the next moment (i.e., the predicted event graph snapshot), which is also the link prediction result.

[0068] In this embodiment, the dynamic graph model includes an encoder structure consisting of a graph neural network (GCN) and a Transformer network, and a decoder structure consisting of a fully connected network.

[0069] An encoder structure can capture the complex relationships between log events and the evolution of complex relationships. By leveraging GCN's advantages in processing static graph data, the local topological features of each event graph snapshot can be captured.

[0070] Specifically, Figure 2 and Figure 3 As shown, the graph neural network can extract temporal features based on each event graph snapshot, each adjacency matrix and each node feature matrix to generate the first feature of each event graph snapshot. ,in, Indicates The first characteristic of an event graph snapshot is that it satisfies the following formula: ; in, , is with Identity matrix of the same dimensions; represents the weight matrix; is the activation function; is the adjacency matrix The degree matrix of .

[0071] Furthermore, the output of the graph neural network (GCN) Input to the variant Transformer network, that is, satisfy the following formula: ; in, Represents the global spatiotemporal feature representation after Transformer network encoding.

[0072] Specifically, the Transformer network can add position encoding to the temporal features of each event graph snapshot to generate the second feature of each event graph snapshot , the position encoding can be generated using sine and cosine functions of different frequencies; then, the Transformer network uses a multi-head attention mechanism to efficiently learn log event features, where the multi-head attention mechanism can be expressed as follows: ; in, Indicates The query matrix of the attention heads; Indicates The query key matrix of the attention heads; represents the dimension of the feature vector (used for scaling dot products); is the activation function; Indicates The output features of an attention head.

[0073] Output of all attention heads After splicing, the third feature is generated and input into the fully connected network. The fully connected network can perform link prediction based on the third feature to generate the final output result Y.

[0074] Specifically, the fully connected network decoder is used to generate a snapshot of the predicted event graph at the next moment; the fully connected network does not directly output a graph , but the output graph The corresponding adjacency matrix , that is, satisfying the formula ,use The predicted event graph snapshot at the next moment is represented, that is, the predicted event graph snapshot at the next moment output by the fully connected network is stored in the computer in the form of an adjacency matrix.

[0075] The log anomaly detection method provided in the embodiment of the present application designs a new dynamic edge flow link prediction method, which can learn the complex relationship between log events based on the constructed dynamic time series event graph and capture the evolution pattern between event relationships; at the same time, it uses the edge information appearing between adjacent event graph snapshots to perform fine-grained updates of node features, thereby mining the dynamic evolution process of event graph snapshots.

[0076] In some embodiments, based on the predicted event graph snapshot, determining whether an abnormality occurs in the log event sequence includes: determining a scoring rule based on the predicted event graph snapshot, and scoring the log event sequence based on the scoring rule to obtain a log sequence score; determining whether the log sequence score is greater than or equal to an abnormality score threshold; if the log sequence score is greater than or equal to the abnormality score threshold, determining that an abnormality occurs in the log event sequence.

[0077] like Figure 3 As shown in Figure 2, we obtain a snapshot of the predicted event graph at the next moment (i.e., moment 𝑡+1) Afterwards, log anomaly detection can be performed based on the results of link prediction, and the predicted event graph snapshot at time 𝑡+1 is The corresponding adjacency matrix can be used express, .

[0078] See also Figure 5 , Figure 5 It is a schematic diagram of the log event sequence scoring process provided in an embodiment of the present application.

[0079] Assumptions Indicates The cth edge of the event sequence The initial score value is Middle Row, No. The data of the column has a value between 0 and 1; according to the adjacency matrix , scoring rules can be formulated to score the log event sequence, and based on this, it can be determined whether anomalies occur in the log event sequence.

[0080] like Figure 5 As shown, assuming the predicted event graph snapshot at time 𝑡+1 The adjacency matrix of Involving 5 nodes, the edge between any two nodes is There is a corresponding initial score value in , for example, the initial score value of the edge between node 1 and node 5 is S (1,5); For each event sequence in the log event sequence , the integrated learning model can be based on The corresponding initial score values ​​are the event sequence Each edge (link) contained in Score and take the minimum of these scores as the event sequence The event sequence score.

[0081] For example Figure 5 For the first event sequence , can be based on The corresponding initial score values ​​are the event sequence Each edge (link) contained in Make a rating.

[0082] Optionally, on the opposite side When scoring, we consider both the global and local perspectives, and comprehensively consider the impact of previously appeared edges (i.e. links) on the edge. From a global perspective, we can consider the edge and all the edges that appeared before it, i.e. .

[0083] Optionally, the global score is calculated as follows: ; in, is the number of the event sequence, The first Strip edge, is the penalty factor, Represents edge exist The initial score value in (between 0 and 1); is the sliding window size.

[0084] Optionally, each event sequence The global score and local score of the event sequence can be obtained by splicing The event sequence score.

[0085] Furthermore, the event sequence scores corresponding to all event sequences are input into the integrated learning model. The integrated learning model can determine the log sequence score of the log event sequence through a voting mechanism (for example, taking the maximum value of the event sequence scores as the log sequence score), and perform anomaly detection based on the log sequence score to determine whether the log sequence score is greater than or equal to the anomaly score threshold.

[0086] If the log sequence score is greater than or equal to the anomaly score threshold, it is determined that the log event sequence is abnormal.

[0087] In some embodiments, based on the predicted event graph snapshot, determining whether an abnormality occurs in the log event sequence includes: determining a scoring rule based on the predicted event graph snapshot, and scoring the log event sequence based on the scoring rule to obtain a log sequence score; determining whether the log sequence score is greater than or equal to an abnormality score threshold; if the log sequence score is less than the abnormality score threshold, determining that no abnormality occurs in the log event sequence.

[0088] The log anomaly detection method provided in the embodiment of the present application utilizes mutual information maximization and spatiotemporal structure embedding methods to generate fine-grained structural representations. The structural representation has the ability to integrate high-order contextual semantic knowledge between explicit and implicit events, and can realize self-supervised log anomaly event detection tasks, meeting the requirements of log anomaly detection tasks guided by few or even zero samples.

[0089] In summary, the log anomaly detection method provided in the embodiment of the present application aims at solving the problems of inefficient log anomaly detection, difficulty in mining implicit event interaction relationships, and easy loss of dynamic evolution relationships. A log anomaly detection method based on dynamic edge flow prediction is proposed: first, the interaction relationship of multi-source log events is constructed using an incremental time series event graph; then, a new dynamic edge flow prediction method is designed to capture the evolutionary pattern between event relationships by learning the complex relationship between log events based on the constructed dynamic time series event graph; finally, the log anomaly detection task is converted into a link prediction task on the time series event graph, and the spatiotemporal features are used to infer the probability of link connection. At the same time, an integrated learning model is used to detect abnormal system logs to capture illegal system operations.

[0090] The present application also provides a log anomaly detection device. Figure 6 , Figure 6 6 is a schematic diagram of the structure of the log anomaly detection device provided in the embodiment of the present application. In the embodiment of the present application, the log anomaly detection device includes a time series event graph construction module 610, a feature update module 620, a link prediction module 630 and an anomaly detection module 640.

[0091] The time series event graph construction module 610 is used to construct a time series event graph based on the log event sequence.

[0092] A time series event graph includes multiple event graph snapshots.

[0093] The feature updating module 620 is used to determine the adjacency matrix of each event graph snapshot and the node feature matrix of each event graph snapshot.

[0094] The adjacency matrix and node feature matrix are used to characterize the spatial characteristics of the event graph snapshot.

[0095] The link prediction module 630 is used to input each event graph snapshot, each adjacency matrix and each node feature matrix into the pre-trained dynamic graph model, perform time series feature extraction and link prediction, and obtain the predicted event graph snapshot output by the dynamic graph model.

[0096] The predicted event graph snapshot is a snapshot of the event graph at a future moment, and is used to characterize potential new relationships between log events.

[0097] The anomaly detection module 640 is used to determine whether an anomaly occurs in the log event sequence based on the predicted event graph snapshot.

[0098] In some embodiments, the log event sequence includes a plurality of event sequences arranged in sequence according to timestamps.

[0099] The time series event graph construction module 610 is used to construct the event graph snapshot of each event sequence in sequence based on the event relationship of each event sequence; an event graph snapshot includes a node set and an edge flow set, the node set includes multiple nodes, the edge flow set includes multiple edges, the nodes are used to represent log events, and the edges are used to represent the execution order of log events; based on each event graph snapshot, a time series event graph is constructed; the multiple event graph snapshots in the time series event graph are arranged in sequence according to timestamps.

[0100] In some embodiments, the feature update module 620 is used to input each event graph snapshot into a pre-trained edge flow model to obtain the adjacency matrix of each event graph snapshot and the node feature matrix of each event graph snapshot output by the edge flow model; wherein the edge flow model is used to update node features according to the edge flow set of adjacent event graph snapshots, and the node feature matrix corresponding to an event graph snapshot is obtained based on the corresponding edge flow set and the initial node feature update, and the initial node feature is the node feature matrix corresponding to the previous event graph snapshot.

[0101] In some embodiments, the dynamic graph model includes a graph neural network, a Transformer network and a fully connected network; the graph neural network is used to extract time series features based on each event graph snapshot, each adjacency matrix and each node feature matrix to generate a first feature of each event graph snapshot; the Transformer network is used to add position encoding to the time series features of each event graph snapshot to generate a second feature of each event graph snapshot; log event feature learning and splicing are performed on the second feature of each event graph snapshot to generate a third feature; the fully connected network is used to perform link prediction based on the third feature to generate a predicted event graph snapshot.

[0102] In some embodiments, the anomaly detection module 640 is used to determine a scoring rule based on the predicted event graph snapshot, and score the log event sequence based on the scoring rule to obtain a log sequence score; determine whether the log sequence score is greater than or equal to the anomaly score threshold; if the log sequence score is greater than or equal to the anomaly score threshold, determine that the log event sequence is abnormal.

[0103] In some embodiments, the anomaly detection module 640 is used to determine a scoring rule based on the predicted event graph snapshot, and score the log event sequence based on the scoring rule to obtain a log sequence score; determine whether the log sequence score is greater than or equal to an anomaly score threshold; if the log sequence score is less than the anomaly score threshold, determine that there is no anomaly in the log event sequence.

[0104] The present application also provides an electronic device, Figure 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the log anomaly detection method.

[0105] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0106] An embodiment of the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the log anomaly detection method provided by the above methods is implemented.

[0107] An embodiment of the present application also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the log anomaly detection method provided by the above methods.

[0108] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A log anomaly detection method, characterized in that: include: Based on the log event sequence, a time series event graph is constructed; The time series event graph includes a plurality of event graph snapshots; Determining an adjacency matrix of each of the event graph snapshots and a node feature matrix of each of the event graph snapshots; The adjacency matrix and the node feature matrix are used to characterize the spatial features of the event graph snapshot; Input each of the event graph snapshots, each of the adjacency matrix, and each of the node feature matrix into a pre-trained dynamic graph model, perform temporal feature extraction and link prediction, and obtain a predicted event graph snapshot output by the dynamic graph model; the predicted event graph snapshot is an event graph snapshot at a future moment, and the predicted event graph snapshot is used to characterize potential new relationships between log events; Based on the predicted event graph snapshot, determine whether the log event sequence is abnormal.

2. The log anomaly detection method according to claim 1, characterized in that: The log event sequence includes a plurality of event sequences arranged in sequence according to timestamps; The step of constructing a time series event graph based on the log event sequence includes: Based on the event relationship of each event sequence, an event graph snapshot of each event sequence is sequentially constructed; one event graph snapshot includes a node set and an edge flow set, the node set includes a plurality of nodes, the edge flow set includes a plurality of edges, the nodes are used to represent log events, and the edges are used to represent the execution order of the log events; Based on each of the event graph snapshots, the time series event graph is constructed; the multiple event graph snapshots in the time series event graph are arranged in sequence according to timestamps.

3. The log anomaly detection method according to claim 2, characterized in that: The determining of the adjacency matrix of each of the event graph snapshots and the node feature matrix of each of the event graph snapshots comprises: Input each of the event graph snapshots into a pre-trained edge flow model to obtain an adjacency matrix of each of the event graph snapshots and a node feature matrix of each of the event graph snapshots output by the edge flow model; Among them, the edge flow model is used to update node features according to the edge flow set of adjacent event graph snapshots, and a node feature matrix corresponding to the event graph snapshot is obtained based on the corresponding edge flow set and the initial node feature update, and the initial node feature is the node feature matrix corresponding to the previous event graph snapshot.

4. The log anomaly detection method according to claim 1, characterized in that: The dynamic graph model includes a graph neural network, a Transformer network and a fully connected network; The graph neural network is used to extract time series features based on each of the event graph snapshots, each of the adjacency matrix and each of the node feature matrix to generate a first feature of each of the event graph snapshots; The Transformer network is used to add position encoding to the temporal features of each of the event graph snapshots to generate a second feature of each of the event graph snapshots; Performing log event feature learning and concatenation on the second feature of each of the event graph snapshots to generate a third feature; The fully connected network is used to perform link prediction based on the third feature and generate the predicted event graph snapshot.

5. The log anomaly detection method according to claim 1, characterized in that: The determining whether the log event sequence is abnormal based on the predicted event graph snapshot includes: Determining a scoring rule based on the predicted event graph snapshot, and scoring the log event sequence based on the scoring rule to obtain a log sequence score; Determining whether the log sequence score is greater than or equal to an anomaly score threshold; If the log sequence score is greater than or equal to the anomaly score threshold, it is determined that an anomaly occurs in the log event sequence.

6. The log anomaly detection method according to claim 1, characterized in that: The determining whether the log event sequence is abnormal based on the predicted event graph snapshot includes: Determining a scoring rule based on the predicted event graph snapshot, and scoring the log event sequence based on the scoring rule to obtain a log sequence score; Determining whether the log sequence score is greater than or equal to an anomaly score threshold; If the log sequence score is less than the anomaly score threshold, it is determined that no anomaly occurs in the log event sequence.

7. A log anomaly detection device, characterized in that: include: The time series event graph construction module is used to construct a time series event graph based on the log event sequence; The time series event graph includes a plurality of event graph snapshots; A feature updating module, used for determining an adjacency matrix of each of the event graph snapshots and a node feature matrix of each of the event graph snapshots; The adjacency matrix and the node feature matrix are used to characterize the spatial features of the event graph snapshot; A link prediction module, used to input each of the event graph snapshots, each of the adjacency matrix and each of the node feature matrix into a pre-trained dynamic graph model, perform temporal feature extraction and link prediction, and obtain a predicted event graph snapshot output by the dynamic graph model; the predicted event graph snapshot is an event graph snapshot at a future moment, and the predicted event graph snapshot is used to characterize potential new relationships between log events; The anomaly detection module is used to determine whether an anomaly occurs in the log event sequence based on the predicted event graph snapshot.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the log anomaly detection method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the log anomaly detection method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the log anomaly detection method according to any one of claims 1 to 6 is implemented.

Images