Log anomaly detection method and system based on local graph attention mechanism

Abstract

The invention relates to a log anomaly detection method and system based on a local graph attention mechanism, and the method comprises the steps: collecting and preprocessing an original log, and generating a log template sequence; constructing a directed graph based on the log template sequence; the attention weight in the directed graph is detected through a graph attention network; reconstructing the log template sequence, and calculating a reconstruction error of the log template sequence; and determining an abnormal result of the log template sequence according to the attention weight and the reconstruction error, and giving an alarm according to the abnormal result. According to the method and the device, the directed graph of the log template sequence is constructed, the attention mechanism is applied, the context dependency relationship between the log templates is explicitly modeled, and the log anomaly is analyzed in combination with the attention weight and the reconstruction error, so that the comprehensiveness of anomaly detection can be improved, false alarms are reduced, and the problem of low anomaly detection accuracy is solved.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a log anomaly detection method and system based on a local graph attention mechanism. Background Technology

[0002] With the widespread adoption of distributed and microservice architectures, system components have become increasingly complex, and logs are massive and heterogeneous. This technological heterogeneity has led to an exponential increase in the complexity of system monitoring and troubleshooting. Runtime logs record various events during system operation, such as operation records and error messages. Effective analysis of these logs can provide crucial technical support for ensuring the reliability and stability of distributed systems.

[0003] Traditional statistical feature-based methods use log template frequency or time-series statistics as features to detect log anomalies through unsupervised learning. However, these methods can only detect numerical anomalies and cannot identify semantic or logical anomalies, resulting in low accuracy. Additionally, sequence model-based methods use LSTM (Long Short-Term Memory) networks or Transformers to model the temporal patterns of log templates and detect anomalies by predicting the next template. However, these methods fail to explicitly model the logical dependencies between log templates, leading to a high false positive rate. Therefore, single statistical or deep learning-based anomaly detection methods struggle to comprehensively learn the relationships between log templates, resulting in low anomaly detection accuracy.

[0004] There is currently no effective solution to the problem of low accuracy in anomaly detection in related technologies. Summary of the Invention

[0005] Therefore, it is necessary to provide a log anomaly detection method and system based on a local graph attention mechanism that can improve the accuracy of anomaly detection, addressing the aforementioned technical problems.

[0006] Firstly, this embodiment provides a log anomaly detection method based on a local graph attention mechanism, including:

[0007] Collect raw logs and preprocess them to generate a log template sequence;

[0008] A directed graph is constructed based on the log template sequence; and the attention weights in the directed graph are detected using a graph attention network.

[0009] The log template sequence is reconstructed, and the reconstruction error of the log template sequence is calculated.

[0010] Based on the attention weight and the reconstruction error, the abnormal results of the log template sequence are determined, and an alarm is issued based on the abnormal results.

[0011] In some embodiments, the process of collecting raw logs and preprocessing them to generate a log template sequence includes:

[0012] The original logs are structured to extract the corresponding log templates and generate an initial log template sequence.

[0013] The initial log template sequence is divided according to a preset window to obtain the log template sequence in each window.

[0014] In some embodiments, detecting attention weights in the directed graph via a graph attention network includes:

[0015] Based on each log template and its neighboring nodes in the directed graph, a feature representation of each log template is generated through the graph attention network; the nodes in the directed graph are log templates, and the edges are the adjacency relationships of the log templates.

[0016] Based on the feature representation of each log template, the attention weight of each edge in the directed graph is calculated.

[0017] In some embodiments, reconstructing the log template sequence and calculating the reconstruction error of the log template sequence includes:

[0018] The log template sequence is input into a temporal convolutional network, and a reconstructed sequence is generated after passing through the dilated convolutional layer in the encoder and the decoder.

[0019] The reconstruction error is calculated based on the reconstruction sequence and the log template sequence.

[0020] In some embodiments, determining the anomalous results of the log template sequence based on the attention weights and the reconstruction error includes:

[0021] The attention weights obtained from pre-training in the directed graph are used as a reference benchmark; and an attention score is calculated based on the attention weights and the reference benchmark.

[0022] The abnormal result is calculated by combining the attention score and the reconstruction error.

[0023] In some embodiments, the method further includes:

[0024] Based on the changing trends of the attention weights and the reconstruction error, the anomaly type of the log template sequence is determined as the anomaly result.

[0025] In some embodiments, the method further includes:

[0026] The exception type and its severity are read through the exception determination interface, and a verification command is sent accordingly.

[0027] Secondly, this embodiment provides a log anomaly detection system based on a local graph attention mechanism, including:

[0028] The acquisition module is used to collect raw logs, preprocess them, and generate log template sequences.

[0029] The graph attention module is used to construct a directed graph based on the log template sequence; and to detect the attention weights in the directed graph through a graph attention network.

[0030] The reconstruction module is used to reconstruct the log template sequence and calculate the reconstruction error of the log template sequence.

[0031] An anomaly detection module is used to determine the abnormal results of the log template sequence based on the attention weight and the reconstruction error, and to issue an alarm based on the abnormal results.

[0032] Thirdly, this embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the log anomaly detection method based on the local graph attention mechanism described in the first aspect above.

[0033] Fourthly, this embodiment provides a storage medium storing a computer program that, when executed by a processor, implements the log anomaly detection method based on the local graph attention mechanism described in the first aspect above.

[0034] Compared with related technologies, the log anomaly detection method and system based on a local graph attention mechanism provided in this embodiment collects and preprocesses raw logs to generate a log template sequence; constructs a directed graph based on the log template sequence; detects the attention weights in the directed graph using a graph attention network; reconstructs the log template sequence and calculates the reconstruction error; determines the anomaly result of the log template sequence based on the attention weights and the reconstruction error, and issues an alarm based on the anomaly result. Through this embodiment, by constructing a directed graph of the log template sequence and applying an attention mechanism, the contextual dependencies between log templates are explicitly modeled. Combining attention weights and reconstruction errors to analyze log anomalies improves the comprehensiveness of anomaly detection and reduces false positives, solving the problem of low accuracy in anomaly detection.

[0035] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0036] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0037] Figure 1 This is a hardware structure block diagram of a terminal for a log anomaly detection method based on a local graph attention mechanism in one embodiment;

[0038] Figure 2 This is a flowchart of a log anomaly detection method based on a local graph attention mechanism in one embodiment;

[0039] Figure 3 This is a flowchart of a log anomaly detection method based on a local graph attention mechanism in another embodiment;

[0040] Figure 4 This is a block diagram of a log anomaly detection system based on a local graph attention mechanism in one embodiment.

[0041] In the diagram: 102, processor; 104, memory; 106, transmission device; 108, input / output device; 10, acquisition module; 20, graph attention module; 30, reconstruction module; 40, anomaly detection module. Detailed Implementation

[0042] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0043] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0044] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for the log anomaly detection method based on local graph attention mechanism in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0045] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the log anomaly detection method based on the local graph attention mechanism in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0046] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0047] This embodiment provides a log anomaly detection method based on a local graph attention mechanism. Figure 2 This is a flowchart of the log anomaly detection method based on the local graph attention mechanism in this embodiment, as follows: Figure 2 As shown, the method includes the following steps:

[0048] Step S201: Collect raw logs and preprocess them to generate log template sequences.

[0049] Specifically, a log collector (such as Fluentd) collects raw logs generated in real time by each server in the distributed system and transmits them to a central processing node. A log parsing tool (such as Drain3) performs structured processing on the raw logs to determine the template ID of each log entry. These template IDs are then arranged chronologically to generate a log template sequence. For example, the log template sequence could be T1 (startup), T2 (authentication), and T3 (execution), where T1, T2, and T3 are the template IDs for different log entries.

[0050] Step S202: Construct a directed graph based on the log template sequence; and detect the attention weights in the directed graph using a graph attention network.

[0051] Specifically, a directed graph is constructed for each log template sequence. In the directed graph, each log template in the sequence is a node, and the adjacency relationship between log templates is an edge. The attention weight of each edge can be calculated from the co-occurrence frequency of log templates during the pre-training stage. For example, if the log template sequence is T1, T2, T3, T1, when constructing the directed graph, T1, T2, and T3 are used as nodes, and directed edges are established based on adjacent nodes in the sequence, such as T1→T2, T2→T3, and T3→T1.

[0052] The Graph Attention Network (GAT) is used to learn the contextual associations and logical relationships between log templates in a directed graph. The attention weights can quantify the semantic association between the log template and its neighboring nodes. Anomaly edge detection is performed by calculating the attention weights of the edges in the directed graph.

[0053] Step S203: Reconstruct the log template sequence and calculate the reconstruction error of the log template sequence.

[0054] Specifically, a temporal convolutional network (TCN) is used to re-encode each log template sequence through an encoder, and then decode it back into a reconstructed sequence through a decoder. The reconstruction error is calculated based on the log template sequence before reconstruction and the reconstructed sequence after reconstruction. The reconstruction error can be either cross-entropy loss or mean squared error, which can reflect the temporal anomalies of the log template sequence.

[0055] Autoencoder structures such as Temporal Convolutional Networks (TCNs) learn during the training phase how to compress (encode) input sequences into low-dimensional features, and then decode and reconstruct them back into the original sequence. For normal log template sequences, TCNs, trained on a large amount of normal temporal data, can efficiently capture their inherent patterns (such as periodicity and dependencies). However, for abnormal log template sequences, because they were not trained or do not conform to the learned patterns, they are difficult to reconstruct accurately, the decoder cannot generate reasonable outputs, or the reconstruction error increases significantly.

[0056] Step S204: Based on the attention weight and reconstruction error, determine the abnormal results of the log template sequence and issue an alarm based on the abnormal results.

[0057] Specifically, attention scores of edges in a directed graph are evaluated based on attention weights. Abnormal results of log template sequences are determined by combining attention scores and reconstruction errors. Abnormal results can be an abnormal score calculated by weighting attention scores and reconstruction errors. An alarm is triggered when the abnormal score exceeds a preset threshold, and the abnormal log context (such as the log template ID and abnormal edge relationships involved) is recorded.

[0058] The above steps construct a directed graph of the log template sequence, and a graph attention network is used to learn the relationships between log templates in the directed graph to obtain attention weights, explicitly modeling the contextual dependencies between log templates. Simultaneously, the log template sequence is reconstructed to obtain reconstruction errors. Log anomaly detection is performed through the complementary synergy of attention weights and reconstruction errors. This not only detects numerical anomalies such as frequency spikes but also accurately identifies logical anomalies such as template disorder and missing elements, significantly improving the comprehensiveness of anomaly detection, reducing false positives, and thus improving the accuracy of anomaly detection. Furthermore, the dual detection mechanism of dynamic attention weight calculation and reconstruction error can significantly improve the response speed to sudden anomaly patterns.

[0059] In some embodiments, step S201 above, which involves collecting and preprocessing raw logs to generate a log template sequence, includes the following steps:

[0060] The original logs are structured, the corresponding log templates are extracted, and an initial log template sequence is generated. The initial log template sequence is then divided according to a preset window to obtain the log template sequence within each window.

[0061] Specifically, the raw logs undergo structured processing to extract corresponding log templates. These template IDs are then arranged chronologically to generate an initial log template sequence. This process eliminates redundant information and reduces the impact of noise on anomaly detection results. In the structured processing, dynamic variables (such as timestamps, IP addresses, and values) in the raw logs are replaced with wildcards to generate static log template IDs. Each log template describes the basic structure and content of the log, representing a specific operation during system operation.

[0062] The continuous initial log template sequence is divided according to a preset window size, and log template sequences within multiple windows are formed with a preset step size. For example, with a preset window size of 10 logs and a preset step size of 1, the log template sequences within each window are T1-T10, T2-T11, T3-T12, etc., with 10 log templates in each window.

[0063] By collecting raw logs and performing structured processing and window segmentation in this embodiment, a log template sequence is obtained. This can eliminate redundant information, reduce the impact of noise on anomaly detection results, and divide long sequences into short windows to better capture and focus on the correlation between short-term logs.

[0064] In some embodiments, step S202 above detects attention weights in a directed graph using a graph attention network, including the following steps:

[0065] Based on each log template and its neighboring nodes in the directed graph, a feature representation of each log template is generated through a graph attention network; the nodes in the directed graph are log templates, and the edges are the adjacency relationships of the log templates; based on the feature representation of each log template, the attention weights of each edge in the directed graph are calculated.

[0066] Specifically, a directed graph is constructed for each log template sequence within each window. Nodes in the directed graph represent log templates, and edges represent the adjacency relationships between log templates. The attention weight of each edge can be calculated from the co-occurrence frequency of log templates during the pre-training phase, thus constructing the directed graph. Specifically, during the pre-training phase, for an edge T1→T2, the total number of times log template T1 appears and the total number of times T2 follows log template T1 are counted. Based on the ratio of the total number of times T2 follows log template T1 to the total number of times log template T1 appears, the collinearity frequency of log templates is calculated, thereby obtaining the attention weight of each edge during the pre-training phase.

[0067] The log template sequence and a directed graph are input into a graph attention network, where an initial feature vector is first generated for each log template. In the graph attention layer, the attention coefficients of each log template and its neighbors are calculated based on the initial feature vectors and then normalized. For each node (log template), the attention coefficients of the log templates of its neighboring nodes are aggregated to finally generate a context-aware feature representation.

[0068] During the inference phase, the final feature representations of each log template in the directed graph are input into the graph attention network, and the attention weights of each edge in the directed graph are calculated. The attention weights quantify the semantic correlation between the log template and its neighboring nodes, and abnormal edge detection is performed by calculating the attention weights of the edges in the directed graph.

[0069] In this embodiment, the graph attention network encodes each log template, dynamically aggregates the feature information of its neighboring nodes, generates a context-aware feature representation, and finally performs abnormal edge detection based on the feature representation, calculates the attention weight of each edge, and can learn the contextual association and logical relationship between log templates in the directed graph through the graph attention network, thereby improving the comprehensiveness of detection.

[0070] In some embodiments, step S203 involves reconstructing the log template sequence and calculating the reconstruction error of the log template sequence, including the following steps:

[0071] The log template sequence is input into a temporal convolutional network, and after passing through the dilated convolutional layer in the encoder and the decoder, a reconstructed sequence is generated. Based on the reconstructed sequence and the log template sequence, the reconstruction error is calculated.

[0072] Specifically, in log anomaly detection based on temporal convolutional networks, the log template sequence is input into the temporal convolutional network. First, the log template sequence within the window is converted into an input format that the model can process. Then, it is input into the temporal convolutional network encoder-decoder structure. The encoder compresses the sequence features to obtain low-dimensional features, and the decoder decodes and restores the reconstructed sequence. The encoder includes dilated convolutional layers, which can further expand the receptive field through multiple layers to capture long-distance temporal dependencies in the log template sequence. The decoder restores the low-dimensional features to the original sequence length and outputs the predicted reconstructed sequence.

[0073] By comparing the differences in the log template at each position in the reconstructed sequence and the log template sequence, the errors at all positions within the window are aggregated to obtain the reconstruction error. The reconstruction error quantifies the reconstruction capability and reflects temporal anomalies in the log template sequence. For normal log template sequences, the temporal convolutional network can reconstruct the sequence well. For abnormal log template sequences, due to not participating in training or not conforming to the learned pattern, accurate reconstruction is difficult, the decoder cannot generate a reasonable output, or the reconstruction error increases significantly. A higher error indicates a greater likelihood of an abnormal input log template sequence.

[0074] In this embodiment, the temporal convolutional network is trained to implicitly learn the statistical patterns of log template sequences. Simultaneously, the network utilizes dilated convolutional layers to capture long-distance temporal dependencies, avoiding the problem of forgetting key log templates in long-sequence scenarios. The reconstruction error is then calculated for log anomaly detection. Furthermore, it complements the detection logic of graph attention networks to improve the comprehensiveness and accuracy of log anomaly detection.

[0075] In some embodiments, the abnormal results of the log template sequence determined in step S204 above based on attention weights and reconstruction errors include the following steps:

[0076] The attention weights obtained from pre-training in the directed graph are used as a reference benchmark; the attention score is calculated based on the attention weights and the reference benchmark; and the abnormal results are obtained by combining the attention score and the reconstruction error.

[0077] Specifically, in anomaly edge detection, the attention weights obtained during the pre-training phase are used as a reference benchmark. Edges in the directed graph exhibiting suspicious anomalies are identified based on the reference benchmark and the attention weights of each edge. A reference range can be set based on the reference benchmark, and edges with attention weights exceeding this range are marked as potentially anomaly-prone. For example, the reference range can be set based on three times the standard deviation of the reference benchmark. Further, an attention score is calculated based on the ratio of the number of edges marked as potentially anomaly-prone to the total number of edges in the directed graph.

[0078] Finally, the attention score and reconstruction error are weighted and summed to obtain the anomaly score of the log template sequence within the window. An alarm is triggered when the anomaly score exceeds a preset threshold, and the anomaly log context (such as the log template ID and anomaly edge relationship involved) and the anomaly score are recorded as an anomaly result.

[0079] By using the complementary collaboration of attention weights and reconstruction errors to perform log anomaly detection in this embodiment, not only can the accuracy of anomaly detection be improved, but the response speed to sudden anomaly patterns can also be significantly enhanced.

[0080] In some embodiments, the method further includes:

[0081] Based on the changing trends of attention weights and reconstruction errors, the anomaly type of the log template sequence is determined as the anomaly result.

[0082] Specifically, common anomaly types include missing critical templates (e.g., "Service Startup → Direct Operation Execution" without a "User Authentication" step), abnormal template order (e.g., "Database Submission → Data Writing" in reverse order), low-frequency template combinations (e.g., a "404 Error" log followed by "Operation Successful" that does not conform to semantics), and timing anomalies in periodic tasks (e.g., a sudden shortening or extension of the interval between scheduled tasks). For each anomaly type, relevant historical log data can be collected, and the trends of related attention weights and reconstruction errors can be summarized as judgment criteria. When an anomaly result is alerted, the corresponding anomaly type can be determined based on the changing trends of attention weights and reconstruction errors, and feedback can be provided accordingly.

[0083] The following example illustrates the detection process for three typical anomaly types:

[0084] (1) Detection of missing key templates

[0085] When the system is running normally, the log template sequence corresponding to the key business process should be [T1 (startup), T2 (authentication), T3 (execution)]. If the sequence [T1, T3] is found to be missing T2, the graph attention network will identify that the attention weight of the "T1→T3" edge is abnormally low (it should normally be T1→T2→T3). At the same time, the reconstruction error of the temporal convolutional network will increase significantly, and the system will determine that the "key step is missing" anomaly and issue an alarm.

[0086] (2) Template sequence anomaly detection

[0087] For a normal data writing process, the log sequence should be [T4 (write), T5 (commit)]. When a reverse sequence [T5, T4] occurs, the graph attention network will detect that the attention weight of the T5→T4 edge is abnormally low. The temporal convolutional network will fail to reconstruct because it has not learned this reverse pattern. The system will determine this as "abnormal template order" and issue an alarm.

[0088] (3) Low-frequency (abnormal) template combination detection

[0089] When a low-frequency template combination that has not appeared during the training phase appears, the graph attention network will identify the abnormal attention weights of the new edge. At the same time, the reconstruction error of the temporal convolutional network exceeds the threshold. The system will judge it as "abnormal low-frequency template combination" and issue an alarm.

[0090] By analyzing the coordinated changing trends of attention weight and reconstruction error in this embodiment, the anomaly type can be quickly identified and repaired simultaneously when an anomaly is detected, significantly improving the operational efficiency of distributed systems.

[0091] In some embodiments, the method further includes:

[0092] The exception type and its severity are read through the exception determination interface, and a verification command is sent accordingly.

[0093] Specifically, this embodiment provides an exception determination interface, which allows the system to return the corresponding exception type and severity level when the user reads the data. The format of the data read is shown in Table 1 below, including 2 bytes: ErrorType indicates the exception type, the character M indicates a missing key template, the character O indicates a sequence exception, and the character C indicates an exception combination. Severity indicates the severity level, with numbers 1-3 representing low, medium, and high severity, respectively. The severity level can be further divided based on the exception score.

[0094] Table 1

[0095]

[0096] Additionally, users can send verification commands containing the operation type and specific operation through the anomaly detection interface, and manually label the abnormal results and types of the alarms. If an anomaly is confirmed, the system records the anomaly type for model optimization; if it is a false alarm, an ignore command is sent and the whitelist is updated. Furthermore, the system can be made to re-collect logs for verification.

[0097] The format for writing is shown in Table 2 below, which includes 2 bytes. Action indicates the operation type, the character V indicates verification, and the character I indicates ignore. Operation indicates the specific operation, the character R indicates the retry process, and the character S indicates skip the check.

[0098] Table 2

[0099]

[0100] In this embodiment, operations and maintenance personnel can label abnormal results of alarms (true anomalies / false alarms), design negative sample generation strategies, and combine dynamic threshold judgment and online incremental learning framework to enable the system to continuously evolve, while ensuring the advantages of unsupervised learning and supporting human feedback optimization.

[0101] The present embodiment will now be described and illustrated through preferred embodiments.

[0102] Figure 3 This is a flowchart of the log anomaly detection method based on the local graph attention mechanism in this embodiment, as follows: Figure 3 As shown, the method includes the following steps:

[0103] Step S301: Collect the raw logs of each server in the distributed system, perform structured processing on the raw logs, extract the corresponding log templates, and generate an initial log template sequence; divide the initial log template sequence according to a preset window to obtain the log template sequence in each window.

[0104] Step S302: Construct a directed graph using the log templates in the log template sequence within each window as nodes and the adjacency relationships between log templates as edges.

[0105] Step S303: Generate a feature representation for each log template using a graph attention network; calculate the attention weights of each edge in the directed graph based on the feature representation of each log template.

[0106] Step S304: Input the log template sequence into the temporal convolutional network, and generate a reconstructed sequence through the dilated convolutional layer in the encoder and the decoder; calculate the reconstruction error based on the reconstructed sequence and the log template sequence.

[0107] Step S305: Use the attention weights obtained from pre-training in the directed graph as a reference benchmark; calculate the attention score based on the attention weights and the reference benchmark; calculate the anomaly score based on the attention score and the reconstruction error; and issue an alarm and report the anomaly type when the anomaly score exceeds a preset threshold.

[0108] This embodiment leverages the synergistic effect of graph attention networks and temporal convolutional networks to not only detect numerical anomalies such as frequency mutations but also accurately identify logical anomalies such as template disorder and missing elements, significantly improving detection comprehensiveness. By constructing a single-sequence directed graph and applying an attention mechanism, the contextual dependencies between log templates are explicitly modeled, effectively solving the high false positive problem caused by traditional methods ignoring template correlations. Furthermore, the use of dilated convolutional structures effectively captures long-window temporal features, avoiding the problem of forgetting key templates in long-sequence scenarios. The introduction of graph attention networks enables the model to directly learn the logical constraints between templates, ensuring compliance verification of business processes. In addition, the dual detection mechanism of dynamic weight calculation and reconstruction error significantly improves the response speed to sudden anomaly patterns.

[0109] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0110] This embodiment also provides a log anomaly detection system based on a local graph attention mechanism. This system is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below can refer to combinations of software and / or hardware that perform a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0111] Figure 4 This is a block diagram of the log anomaly detection system based on the local graph attention mechanism in this embodiment, as shown below. Figure 4 As shown, the system includes:

[0112] The acquisition module 10 is used to collect raw logs and preprocess them to generate a log template sequence;

[0113] Graph attention module 20 is used to construct a directed graph based on log template sequences and to detect attention weights in the directed graph through a graph attention network.

[0114] The reconstruction module 30 is used to reconstruct the log template sequence and calculate the reconstruction error of the log template sequence.

[0115] The anomaly detection module 40 is used to determine the abnormal results of the log template sequence based on the attention weight and reconstruction error, and to issue an alarm based on the abnormal results.

[0116] The system provided in this embodiment constructs a directed graph of the log template sequence and uses a graph attention network to learn the relationships between log templates in the directed graph to obtain attention weights, explicitly modeling the contextual dependencies between log templates. Simultaneously, the log template sequence is reconstructed to obtain reconstruction errors. Log anomaly detection is performed based on the complementary synergy of attention weights and reconstruction errors. This not only detects numerical anomalies such as frequency mutations but also accurately identifies logical anomalies such as template disorder and missing elements, significantly improving the comprehensiveness of anomaly detection, reducing false positives, and thus improving the accuracy of anomaly detection. Furthermore, the dual detection mechanism of dynamic attention weight calculation and reconstruction error can significantly improve the response speed to sudden anomaly patterns.

[0117] In some embodiments, the acquisition module 10 is further configured to:

[0118] The original logs are structured, the corresponding log templates are extracted, and an initial log template sequence is generated. The initial log template sequence is then divided according to a preset window to obtain the log template sequence within each window.

[0119] In some embodiments, the graph attention module 20 is also used for:

[0120] Based on each log template and its neighboring nodes in the directed graph, a feature representation of each log template is generated through a graph attention network; the nodes in the directed graph are log templates, and the edges are the adjacency relationships of the log templates; based on the feature representation of each log template, the attention weights of each edge in the directed graph are calculated.

[0121] In some embodiments, the reconstruction module 30 is further configured to:

[0122] The log template sequence is input into a temporal convolutional network, and after passing through the dilated convolutional layer in the encoder and the decoder, a reconstructed sequence is generated. Based on the reconstructed sequence and the log template sequence, the reconstruction error is calculated.

[0123] In some embodiments, the anomaly detection module 40 is further configured to:

[0124] The attention weights obtained from pre-training in the directed graph are used as a reference benchmark; the attention score is calculated based on the attention weights and the reference benchmark; and the abnormal results are obtained by combining the attention score and the reconstruction error.

[0125] In some embodiments, the anomaly detection module 40 is further configured to:

[0126] Based on the changing trends of attention weights and reconstruction errors, the anomaly type of the log template sequence is determined as the anomaly result.

[0127] In some embodiments, the anomaly detection module 40 is further configured to:

[0128] The exception type and its severity are read through the exception determination interface, and a verification command is sent accordingly.

[0129] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0130] This embodiment also provides a computer device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0131] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0132] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0133] Furthermore, in conjunction with the log anomaly detection method based on local graph attention mechanism provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the log anomaly detection methods based on local graph attention mechanism in the above embodiments.

[0134] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0135] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0136] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or alternative to other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0137] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A log anomaly detection method based on a local graph attention mechanism, characterized in that, include: Collect raw logs and preprocess them to generate a log template sequence; Construct a directed graph based on the log template sequence; The attention weights in the directed graph are detected using a graph attention network. The log template sequence is reconstructed, and the reconstruction error of the log template sequence is calculated. Based on the attention weight and the reconstruction error, the abnormal results of the log template sequence are determined, and an alarm is issued based on the abnormal results.

2. The log anomaly detection method based on local graph attention mechanism according to claim 1, characterized in that, The process of collecting and preprocessing raw logs to generate a log template sequence includes: The original logs are structured to extract the corresponding log templates and generate an initial log template sequence. The initial log template sequence is divided according to a preset window to obtain the log template sequence in each window.

3. The log anomaly detection method based on local graph attention mechanism according to claim 1, characterized in that, The step of detecting attention weights in the directed graph using a graph attention network includes: Based on each log template and its neighboring nodes in the directed graph, a feature representation of each log template is generated through the graph attention network; the nodes in the directed graph are log templates, and the edges are the adjacency relationships of the log templates. Based on the feature representation of each log template, the attention weight of each edge in the directed graph is calculated.

4. The log anomaly detection method based on local graph attention mechanism according to claim 1, characterized in that, The process of reconstructing the log template sequence and calculating the reconstruction error of the log template sequence includes: The log template sequence is input into a temporal convolutional network, and a reconstructed sequence is generated after passing through the dilated convolutional layer in the encoder and the decoder. The reconstruction error is calculated based on the reconstruction sequence and the log template sequence.

5. The log anomaly detection method based on local graph attention mechanism according to claim 1, characterized in that, The step of determining the abnormal results of the log template sequence based on the attention weight and the reconstruction error includes: The attention weights obtained from pre-training in the directed graph are used as a reference benchmark; and an attention score is calculated based on the attention weights and the reference benchmark. The abnormal result is calculated by combining the attention score and the reconstruction error.

6. The log anomaly detection method based on local graph attention mechanism according to claim 1, characterized in that, The method further includes: Based on the changing trends of the attention weights and the reconstruction error, the anomaly type of the log template sequence is determined as the anomaly result.

7. The log anomaly detection method based on local graph attention mechanism according to claim 6, characterized in that, The method further includes: The exception type and its severity are read through the exception determination interface, and a verification command is sent accordingly.

8. A log anomaly detection system based on a local graph attention mechanism, characterized in that, include: The acquisition module is used to collect raw logs, preprocess them, and generate log template sequences. The graph attention module is used to construct a directed graph based on the log template sequence; The attention weights in the directed graph are detected using a graph attention network. The reconstruction module is used to reconstruct the log template sequence and calculate the reconstruction error of the log template sequence. An anomaly detection module is used to determine the abnormal results of the log template sequence based on the attention weight and the reconstruction error, and to issue an alarm based on the abnormal results.

9. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the log anomaly detection method based on the local graph attention mechanism according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the log anomaly detection method based on the local graph attention mechanism as described in any one of claims 1 to 7.

Images