Log anomaly detection system based on enhanced dynamic graph and interpretable diagnosis

Abstract

The invention belongs to but is not limited to the technical field of log anomaly detection, and discloses a log relationship anomaly detection system which fuses a large language model (LLM) and a dynamic graph neural network and has interpretable and diagnostic capabilities. The method comprises the following steps: collecting an original log stream from a distributed system, carrying out timestamp alignment, denoising and block preprocessing, carrying out field-level named entity recognition on a log template and parameters by adopting a pre-trained large language model, and generating deep semantic embedding; constructing a continuous time dynamic graph according to the log event, the field instance and the time relationship thereof, and supporting 0-hop, 1-hop and multi-hop relationship modeling; carrying out joint coding on a graph structure and time sequence dependence by fusing GCN and Set-Transform, relieving data imbalance by adopting category-guided Mixup and comparative learning, and outputting an edge-level abnormal score; and organizing the abnormal edge / subgraph and the related context into a structured Prompt, and calling LLM to generate natural language interpretation and root cause suggestions.

Description

Technical Field

[0001] This invention belongs to, but is not limited to, the field of log anomaly detection technology, and particularly relates to a log relationship anomaly detection system that integrates a large language model (LLM) and a dynamic graph neural network and has interpretable diagnostic capabilities. Background Technology

[0002] System logs are the cornerstone of understanding the behavior of modern, complex IT systems and ensuring their stability and security. Logs record a wealth of runtime information, which is crucial for ensuring system reliability, security, and troubleshooting. However, existing log anomaly detection methods face the following main challenges when dealing with the complexity of modern systems:

[0003] (1) Insufficient semantic understanding: Traditional methods and some existing graph methods have a relatively superficial understanding of the semantics of logs. Their node or edge representations are usually based on simple statistical features or general embeddings, which may not be able to capture the deep semantics and subtle differences of logs in specific domains.

[0004] (2) Incomplete relationship capture: Sequence-based deep learning methods (such as DeepLog and LogBERT) usually treat logs as independent sequences, making it difficult to capture complex dependencies across different entities or log sources. Many critical failures are not caused by a single event, but are reflected in abnormal interaction patterns between multiple entities.

[0005] (3) Data imbalance problem: In real log data, the proportion of abnormal logs is much lower than that of normal logs. This common class imbalance problem poses a challenge to the detection performance of the model.

[0006] (4) Lack of interpretability: Most methods, especially those based on graph neural networks (GNNs), are like "black boxes". Even if an anomaly is detected, it is difficult to automatically provide a clear, human-understandable explanation, and cannot explain the cause of the anomaly or its possible root cause, thus limiting its practical application value.

[0007] The problem with existing technology is:

[0008] In traditional log anomaly detection systems, after operations and maintenance personnel hand over log data to the detection model, the model's understanding of log semantics remains at a shallow statistical or general embedding level, making it difficult to capture domain-specific semantics and subtle differences, resulting in the missed detection of critical anomalies. At the same time, sequential deep models treat logs as independent sequences, losing control over complex dependencies across entities, which is not conducive to discovering faults caused by multi-entity interactions. Once anomalies spread, they may endanger the stability and security of the entire system.

[0009] Furthermore, traditional methods often focus on binary classification of overall anomalies. The datasets are stored centrally and are extremely imbalanced, which may lead to performance bottlenecks and internal biases due to insufficient rare anomaly samples. These issues include, but are not limited to, a surge in false positive rates and missed detection of key anomalies, and it is difficult to provide an explainable root cause.

[0010] Current research has attempted to introduce graph neural networks (GNNs) to model relationships between logs. However, existing models simply utilize the decentralization capabilities of graph structures to enhance relationship capture and use the aggregation mechanism of GNNs to store log relationship information to ensure that they can express complex interactions. They do not consider the accurate characterization of the deep semantics of logs, the effective handling of extremely imbalanced data, or the high requirement for human-understandable interpretation of the detection results.

[0011] Based on the above analysis, the urgent technical problems that need to be solved in the existing technology are:

[0012] (1) Problem of capturing deep semantics of logs: Existing technologies (traditional statistical methods, partial graph methods) only understand log semantics based on simple statistical features or general embeddings, which cannot capture the deep semantics and subtle differences of logs in specific domains, and are prone to missing key anomalies.

[0013] (2) Problem of capturing complex dependencies across entities: Existing sequential deep learning methods (such as DeepLog and LogBERT) treat logs as independent sequences, and existing GNN-type methods simply use graph structures to enhance relationship capture. Neither can fully capture complex dependencies across different entities / log sources, making it difficult to discover faults caused by multi-entity interactions.

[0014] (3) Problem of handling log data imbalance: Existing technologies mostly focus on binary classification for anomaly detection, and have not effectively addressed the problem of data imbalance in real logs where the proportion of abnormal samples is much lower than that of normal samples, leading to model performance bottlenecks (such as a surge in false alarm rate and missed detection of key anomalies) and internal biases.

[0015] (4) Interpretability of detection results: Existing technologies (especially GNN models) are mostly "black boxes". Even if an anomaly is detected, they cannot automatically provide an explanation that humans can understand (such as the cause of the anomaly or the root cause), which limits the practical application value. Summary of the Invention

[0016] To address the problems existing in the prior art, this invention provides a log anomaly detection system (LLM-GLAD) based on Large Language Model (LLM) enhanced dynamic graph relation learning and interpretable diagnosis. This invention utilizes LLM for log field extraction, semantic enhancement, and data augmentation, and proposes a method (or mechanism) for multi-scale continuous-time dynamic graph modeling. This invention employs an LLM-based interpretable module that can provide natural language interpretation and potential root cause analysis suggestions.

[0017] The present invention is implemented as follows: a log anomaly detection system based on enhanced dynamic graphs and interpretable diagnostics, comprising a parsing and extraction module, a dynamic graph construction module, a graph anomaly detection module, and a diagnostic and analysis module;

[0018] The parsing and extraction module extracts LLM-enhanced log parsing and fields, collects raw log streams from the distributed system, performs preprocessing such as timestamp alignment, noise reduction, and block segmentation, and uses a pre-trained large language model (such as BART / Qwen) to perform field-level named entity recognition (NER) on log templates and parameters, and generates deep semantic embeddings.

[0019] The dynamic graph construction module builds multi-scale continuous-time dynamic graphs based on log events, field instances, and their temporal relationships, supporting 0-hop, 1-hop, and multi-hop relationship modeling.

[0020] The graph anomaly detection module integrates GCN and Set-Transformer to jointly encode graph structure and temporal dependencies, and uses category-guided Mixup and contrastive learning to alleviate data imbalance, and outputs edge-level anomaly scores.

[0021] The diagnosis and analysis module performs interpretable anomaly diagnosis and root cause analysis, organizes the abnormal edges / subgraphs and related contexts into a structured Prompt, and calls LLM to generate natural language explanations and root cause suggestions.

[0022] Furthermore, the LLM-enhanced log parsing and field extraction include:

[0023] S11, use the Drain parser to process the raw logs of BGL to obtain the log template and parameters;

[0024] S12, using the finely tuned Qwen1.5 model, extracts fields from the parameters and identifies key fields such as node ID and IP address;

[0025] S13, For each log template and field, use Qwen1.5 to generate its initial semantic embedding;

[0026] S14, Apply topic-aware semantic enhancement method to the log template: Use LDA to extract topic terms to enhance the log template L T The expression .

[0027] Furthermore, the topic-aware coding formula is used to calculate the w of each word. i The final vector v in template LT i The formula takes into account both the word and all the keywords in the template. j The relative positional distance d of (j∈p, p is the set of keyword indices)ij (used to preserve word order information)

[0028] d ij =1 / (1+e) |i-j| ),

[0029] and semantic similarity Cos ij

[0030]

[0031] For non-topic terms w i Its final vector v i It is the result of its initial vector being weighted and adjusted by the two constraint factors mentioned above.

[0032]

[0033] Where m is the number of keywords, and λ is a hyperparameter that balances position and semantic weights.

[0034] Calculate the refined vector {v1, v2, ..., v} of each word in the template. t Regarding the keywords themselves, The final aggregation yields the final feature vector V of the template node. template The node feature vector V for each log event template. template And the node feature vector V of each extracted key field instance. field (Mainly derived from direct LLM embedding), together they constitute "NodeFeature Vectors".

[0035] Furthermore, the construction of the multi-scale continuous-time dynamic graph includes:

[0036] S21, use the log template and extracted key fields as graph nodes;

[0037] S22, Based on the timestamps and relationships of log events, construct dynamic graph sequences at three scales: h = 0, 1, > 1;

[0038] S23, Calculate Semantic Similarity (SS): Calculate the semantic similarity between two connected nodes (v... i ,v j The cosine similarity between vectors is:

[0039]

[0040] S24, Calculate the Time Interval (TI): Calculate the difference in timestamps between the two connected event nodes.

[0041] TI(v i ,v j ) = timestamp j -timestamp i .

[0042] Furthermore, the graph anomaly detection includes:

[0043] S31. Input the constructed multi-scale graph sequence into the GCN+SetTransformer encoder. Set the number of GCN layers to 3 and the hidden unit dimension to 768.

[0044] S32, during training, uses a learning rate of 1e -3 The weight decays to 5e -7 .

[0045] S33 applies category-guided Mixup to augment data on normal and anomalous samples at the hidden level.

[0046] S34, calculate the total loss using the following formula.

[0047] L = L e +αL g +β(L cls +δL con )+Regularization,

[0048] Where α, β, δ are hyperparameters, L e For edge-level loss, L g For graph-level loss, L cls For classification loss, L con To contrast the loss, Regularization is the L2 regularization term for the model parameters. By minimizing this total loss, the model can simultaneously learn to distinguish between normal / abnormal edges, cluster normal graphs, and utilize data augmentation to improve generalization ability.

[0049] The aforementioned multi-granularity loss function is used for model optimization. This loss function integrates the objectives of edge-level, graph-level, classification, and contrastive learning.

[0050] S35 outputs the anomaly detection results.

[0051] Furthermore, the explainable abnormal diagnosis and root cause analysis include:

[0052] S41, when the model detects an abnormal edge (e.g., a connection from node A to node B that should not exist), the system collects the original log text related to the edge, the features of nodes A and B, the features of the edge, and the abnormal score.

[0053] S42 integrates this information into a Prompt, for example: "You are a supercomputer operations expert. An abnormal correlation has been detected between node A (log event: 'kernel error') and node B (IP address: XXXX), with the time interval being abnormally shortened. Please analyze this anomaly and speculate on the possible causes."

[0054] S43. Input this prompt into the Qwen1.5 model. The model generates the following report:

[0055] Explanation of the anomaly: "The association between the detected 'kernel error' log and a specific IP address is extremely rare in normal mode, and the time interval between occurrences is much shorter than the historical average. This suggests that a sudden hardware or software failure may have occurred, rather than normal system operation."

[0056] Root cause recommendations: "1. Check the hardware health status of the compute node with IP address XXXX. 2. Review the detailed system logs related to kernel errors on this node. 3. Check for recent software updates or configuration changes."

[0057] Another objective of this invention is to provide a log anomaly detection method for a log anomaly detection system based on large language model-enhanced dynamic graph relation learning and interpretable diagnostics, comprising the following steps:

[0058] Step 1, LLM-enhanced log parsing and field extraction: Real-time log stream is pulled, parsing and cleaning are completed, and LLM is used to complete field extraction and semantic embedding generation;

[0059] Step 2, Construction of a multi-scale continuous-time dynamic graph: Insert nodes and edges based on event timestamps and field relationships;

[0060] Step 3, Graph Anomaly Detection: Run the GCN-Transformer encoder to calculate edge anomaly scores and mark suspicious relationships;

[0061] Step 4, Explainable Anomaly Diagnosis and Root Cause Analysis: Input the anomaly context into LLM to generate natural language explanations and root cause suggestions. Push the anomaly report and explanation to the operation and maintenance platform and support closed-loop optimization with human feedback.

[0062] Another object of the present invention is to provide a computer device, the computer device including a memory and a processor, the memory storing a computer program, and when the computer program is executed by the processor, causing the processor to perform the steps of the log anomaly detection method based on large language model-enhanced dynamic graph relation learning and interpretable diagnosis.

[0063] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the log anomaly detection method based on large language model-enhanced dynamic graph relation learning and interpretable diagnosis.

[0064] Another objective of this invention is to provide an information data processing terminal, which includes the aforementioned log anomaly detection system based on large language model-enhanced dynamic graph relationship learning and interpretable diagnosis.

[0065] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0066] First, this invention addresses the four core technical challenges in existing log anomaly detection technologies by providing a systematic solution through innovative technical approaches, achieving significant technical results:

[0067] (1) This solves the problem of insufficient deep semantic capture of logs:

[0068] Existing technologies often rely on shallow statistical features or general word embeddings to understand logs, failing to capture subtle semantic differences specific to certain domains, leading to missed detections of critical anomalies. This invention innovatively introduces a Large Language Model (LLM) throughout the entire log processing workflow. In the parsing phase, a finely tuned Qwen1.5 model is used for field-level Named Entity Recognition (NER) and initial semantic embedding generation, surpassing the capabilities of traditional parsers. Furthermore, a unique topic-word-aware semantic enhancement method is developed, refining word vector representations by combining the position and semantic similarity of topic words, enabling node features to accurately reflect their deeper meaning within specific business scenarios. This shift from "general understanding" to "deep perception" is fundamental to solving the problem of missed detections of critical anomalies, and its effectiveness has been validated in experiments on datasets such as BGL.

[0069] (2) This solves the problem of incomplete capture of complex cross-entity dependencies:

[0070] Traditional sequence models (such as DeepLog) treat logs as isolated sequences, while existing graph methods have a rudimentary utilization of relationships, making it difficult to detect hidden faults caused by abnormal interactions among multiple entities. This invention proposes and constructs a multi-scale continuous-time dynamic graph capable of simultaneously modeling system dependencies at 0-hop (instantaneous state), 1-hop (direct interaction), and multi-hop (indirect impact) levels. This approach overcomes the limitations of fixed time windows or single-relationship scales by fusing GCN and Set-Transformer to jointly encode the structural information and temporal dependencies of the graph, thereby comprehensively and dynamically characterizing the interaction patterns between entities in complex IT systems. This enables the invention to effectively discover and locate complex, cross-entity abnormal patterns composed of multiple seemingly normal events.

[0071] (3) This solved the problem of extreme imbalance in log data categories:

[0072] In real-world operational scenarios, anomaly logs account for a very small percentage, making model training difficult and prone to high false positive rates or missed detections of critical anomalies. This invention goes beyond simple binary classification detection, introducing a category-guided Mixup data augmentation strategy and multi-granularity loss functions into the graph anomaly detection module. By mixing normal and anomaly samples in the latent space, high-quality pseudo-samples are generated, effectively expanding the sparse anomaly class data. Simultaneously, the designed total loss function integrates multiple objectives such as edge-level, graph-level, classification, and contrastive learning, forcing the model to learn the boundaries of normal patterns at different granularities. This significantly improves the model's robustness and generalization ability under imbalanced data, effectively reducing false positive and false negative rates.

[0073] (4) It solves the problem of lack of interpretability of test results:

[0074] Especially for GNN-based models, their "black box" nature has long been a problem plaguing the industry. Even when anomalies are detected, operations and maintenance personnel cannot understand the cause of the anomaly, greatly limiting their practical application value. This invention creatively designs an interpretable anomaly diagnosis and root cause analysis module based on LLM. This module automatically constructs a structured prompt from the detected abstract graph anomalies (such as an abnormal edge or subgraph) and their contextual information, and calls LLM to generate human-readable natural language explanations and specific root cause investigation suggestions. This is not just about simply reporting anomalies, but also about providing decision support on "why the anomaly occurred" and "what should be done," combining the powerful relationship discovery capabilities of GNNs with the superior cognitive reasoning capabilities of LLM, fundamentally solving the "black box" problem and greatly improving the practicality and operability of the detection system.

[0075] Second, the technical solution of this invention fills a technological gap in the industry both domestically and internationally:

[0076] Currently, technical solutions in the field of log anomaly detection are showing a "polarization" trend: on one end are complex models that pursue high-precision detection, such as various deep learning and graph neural network models, but they generally lack interpretability and are "black boxes" that are difficult for operation and maintenance personnel to trust; on the other end are traditional statistical or rule-based methods that have a certain degree of interpretability, but their semantic understanding ability is weak and they are difficult to cope with the complexity and dynamism of modern IT systems.

[0077] For a long time, there has been a lack of an end-to-end solution that can seamlessly integrate "precise detection" with "intelligent diagnostic interpretation" in both domestic and international markets and research fields.

[0078] This invention is the first to deeply integrate the capabilities of Large Language Models (LLMs) into the entire lifecycle of log anomaly detection—from initial semantic enhancement to intermediate data enhancement, and finally to interpretable diagnosis and root cause analysis. This fills the huge gap between "detection" and "diagnosis" in existing technologies. This invention no longer simply outputs a cold "anomaly" label or score, but constructs a complete automated intelligent operation and maintenance closed loop of "perception-detection-diagnosis-suggestion". This systematic and collaborative combination of the powerful relational modeling capabilities of GNNs and the cognitive reasoning capabilities of LLMs to achieve high-precision, high-reliability, and high-availability log anomaly detection is unprecedented in existing public technologies, thus filling a key technological gap in this field. (3) The technical solution of this invention solves a technical problem that people have long wanted to solve but have never been able to achieve: the crisis of credibility and interpretability of complex deep learning models in the IT operation and maintenance field.

[0079] Over the years, researchers have tried various methods to explain the decisions of deep learning models, such as attention mechanisms and SHAP value analysis. However, these methods often only provide low-level clues such as "feature importance" and cannot generate high-level, actionable diagnostic reports that align with the thinking patterns of operations and maintenance experts. What operations and maintenance personnel truly need are clear answers to "why it happened, the scope of its impact, and how to solve it," not just mathematical attributions.

[0080] This invention ingeniously solves this problem. Instead of attempting to "open" the "black box" of GNN, it innovatively uses its "output" as a "bridge" to communicate with another powerful tool—Large Language Model (LLM). By transforming the abnormal structures detected by GNN (such as abnormal edges and related subgraphs) and their contextual information into a structured, expert-oriented prompt, this invention successfully transforms a complex graph data science problem into a natural language generation and domain knowledge reasoning problem that LLM excels at. This is equivalent to equipping GNN with a "simultaneous interpreter" and a "domain expert advisor," enabling it to translate and elevate the machine-readable anomaly patterns it discovers into operational insights that human experts can understand and adopt in real time. This "GNN+LLM" collaborative diagnostic paradigm successfully overcomes the long-standing problem of deep model interpretability, making it possible to implement high-precision automated anomaly detection systems in real-world production environments. Attached Figure Description

[0081] Figure 1 This is a structural diagram of the log anomaly detection system based on large language model-enhanced dynamic graph relationship learning and interpretable diagnosis provided in an embodiment of the present invention.

[0082] Figure 2 This is a flowchart of LLM-enhanced log parsing and field extraction provided in an embodiment of the present invention.

[0083] Figure 3 This is a flowchart of the multi-scale continuous-time dynamic graph construction process provided in the embodiments of the present invention.

[0084] Figure 4 This is a flowchart of the anomaly detection process provided in an embodiment of the present invention.

[0085] Figure 5 This is a flowchart of interpretable abnormality diagnosis and root cause analysis provided in the embodiments of the present invention.

[0086] Figure 6 This is a line graph showing the comparison results of different frameworks provided in this embodiment of the invention on the BGL dataset.

[0087] Figure 7 This is a line graph showing the comparison results of different frameworks provided in this embodiment of the invention on the HDFS dataset.

[0088] Figure 8 This is a line graph showing the comparison results of different frameworks provided in this embodiment of the invention on the Thunderbird dataset.

[0089] Figure 9 This is a line graph showing the ablation experimental results of this invention on various datasets. Detailed Implementation

[0090] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0091] In the current industrial environment, distributed systems and cloud-native architectures have been widely applied in critical industries such as finance, e-commerce, energy, and telecommunications. With increasing system complexity, log data volumes are exploding, making traditional rule-based anomaly detection methods insufficient for real-time analysis and diagnosis of multi-source heterogeneous log streams. Existing technologies often can only locate single-point anomalies, lacking the ability to model complex dependencies across nodes and time windows. This leads to a significant decrease in detection rate and accuracy when dealing with link-level failures, covert attacks, or rare anomalies, failing to meet the operational needs of large-scale distributed scenarios. This technical problem in industrial applications is prominent in terms of operational costs, downtime losses, and security risks, thus urgently requiring new intelligent solutions.

[0092] This invention overcomes the reliance of existing technologies on template matching and simple field extraction by introducing Large Language Model (LLM)-enhanced log parsing. It leverages pre-trained models to introduce deep semantic embeddings into log parsing, enabling log data to possess not only structured features but also semantic connections across events. This approach effectively addresses the challenges of diverse log formats, complex field meanings, and ambiguous contextual relationships, making the parsing and feature extraction processes more intelligent and robust, and providing semantic support for subsequent graph modeling.

[0093] In the dynamic graph construction phase, the system generates a multi-scale continuous-time dynamic graph through joint constraints of timestamps and event fields. This considers both direct dependencies under zero-hop adjacency relationships and long-range semantic transfers caused by one-hop and multi-hop relationships. This approach can reflect the temporal dynamics and structural evolution characteristics between log events, making up for the shortcomings of static graphs and sequence models in capturing complex dependency patterns. It is particularly suitable for cloud-native systems with frequent interactions between multiple nodes and services.

[0094] In the anomaly detection phase, this invention utilizes joint encoding of GCN and Set Transformer to deeply fuse the topological dependencies and temporal features of the graph structure. Compared to traditional detection methods based on statistical thresholds or single deep learning models, this design can uniformly model multi-granularity anomalies (edge-level, subgraph-level, and graph-level). Furthermore, the introduction of category-guided mixup and contrastive learning mechanisms effectively alleviates the training bottleneck caused by the scarcity of anomaly samples and data imbalance, improving the model's generalization ability in industrial-grade data environments.

[0095] This invention achieves interpretable output in the diagnosis and analysis stages. The system organizes detected anomalies and related contexts into a structured prompt input large language model, automatically generating natural language explanations and root cause analysis suggestions. This not only improves the transparency of anomaly detection but also shortens manual troubleshooting time and improves the decision-making efficiency of operations and maintenance personnel. Compared with existing "black box" anomaly detection models, this method significantly enhances the model's practicality and reliability.

[0096] This invention constructs a closed-loop log anomaly detection system through LLM-enhanced semantic parsing → multi-scale dynamic graph modeling → joint detection using graph neural networks and Transformers → LLM-generated interpretable diagnostics. This closed loop not only achieves the linkage between detection, diagnosis, and feedback, but also supports the continuous optimization of anomaly handling strategies by the operations and maintenance team, possessing high industrial application value.

[0097] This embodiment uses the application on the BGL supercomputer log dataset as an example to illustrate the specific process of the present invention.

[0098] Experimental environment: The server was configured with a 16-core CPU, an NVIDIA RTX 4090 (24GB) GPU, and 120GB of RAM. Python and the PyTorch Geometric framework were used. Qwen 1.5 was selected for the large language model. Dataset: The BGL dataset contains approximately 4.74 million log messages, of which anomalous data accounts for approximately 7.34%.

[0099] The BGL dataset format is: 2023-10-05 14:23:56.789 ERROR node-042 storage98761234561 Disk / dev / sda1 read error: sector 123456, I / O timeout

[0100] Parsing of fields in the example: Log timestamp: 2023-10-05 14:23:56.789 → October 5, 2023, 14:23:56.789. Log type identifier: ERROR → This log is an error event. Node / host ID: node-042 → The log comes from HPC node number 042. Component / module name: storage → The event is associated with the storage module (disk). Process ID (PID): 9876 → The process ID that triggered this event is 9876 (possibly a disk read / write process). Log sequence number: 123456 → The unique sequence number of this log in the dataset. Fault label: 1 → This log is a fault-related log (disk read error). Log description: Disk / dev / sda1 read error: sector 123456, I / O timeout → Specific event: A read error occurred in sector 123456 of disk / dev / sda1, and I / O timed out.

[0101] like Figure 1 As shown, this embodiment of the invention provides a log anomaly detection system based on enhanced dynamic graphs and interpretable diagnostics, including: an LLM-enhanced log parsing and field extraction module, a multi-scale continuous-time dynamic graph construction module, an enhanced graph representation learning and anomaly detection module, and an interpretable anomaly diagnosis and root cause analysis module.

[0102] The LLM-enhanced log parsing and field extraction module is used to collect raw log streams from distributed systems, perform preprocessing such as timestamp alignment, noise reduction, and block segmentation, and use pre-trained large language models (such as BART / Qwen) to perform field-level named entity recognition (NER) on log templates and parameters, and generate deep semantic embeddings.

[0103] The multi-scale continuous-time dynamic graph construction module builds continuous-time dynamic graphs based on log events, field instances and their temporal relationships, supporting 0-hop, 1-hop and multi-hop relationship modeling.

[0104] The graph anomaly detection module integrates GCN and Set-Transformer to jointly encode graph structure and temporal dependencies, and uses category-guided Mixup and contrastive learning to alleviate data imbalance, and outputs edge-level anomaly scores.

[0105] The Explainable Anomaly Diagnosis and Root Cause Analysis module organizes anomaly edges / subgraphs and related contexts into a structured Prompt, and calls LLM to generate natural language explanations and root cause suggestions.

[0106] In the log parsing phase, the system uses a pre-trained large language model (such as BART or Qwen) to perform semantic enhancement parsing on the raw logs. After performing timestamp alignment, noise reduction, and block preprocessing on the log stream, this module abstracts the log statements into templates and parameters, and uses named entity recognition technology to accurately annotate key fields, such as IP addresses, error codes, and thread IDs, thereby generating deep semantic embeddings. This process ensures that log data can be used by subsequent modules in a unified, structured, and semantically rich form.

[0107] The system constructs a multi-scale continuous-time dynamic graph, modeling log events and field instances as graph nodes and connecting them with temporal and interaction relationships as edges. The system can represent not only zero-hop co-event dependencies but also one-hop or multi-hop cross-event dependencies, thus capturing the temporal and causal relationships in log data from near to far. This continuous-time dynamic graph structure enables the system to model the evolution of log behavior at both fine-grained and macroscopic levels.

[0108] In the graph representation and anomaly detection stages, the system combines Graph Convolutional Networks (GCNs) and Set-Transformer structures to jointly encode graph topological information and temporal dependencies. To address the data imbalance problem commonly found in log anomaly detection, the system introduces category-guided Mixup enhancement and contrastive learning mechanisms, enabling the model to learn robust discrimination boundaries. Finally, the model outputs anomaly scores for each edge or subgraph, achieving accurate identification of potential anomaly patterns.

[0109] In the anomaly explanation and root cause analysis phase, the system organizes the detected abnormal edges or subgraphs and their contextual information into structured prompts and calls a large language model for reasoning. The model not only generates anomaly explanations in natural language but also proposes possible root cause analyses and troubleshooting suggestions, enabling operations and maintenance personnel to quickly understand the source of the anomaly and avoid relying solely on black-box detection results.

[0110] The system's overall operation mechanism embodies a closed loop of "detection-diagnosis-interpretation": the front-end module is responsible for semantic parsing of the raw log stream, the middle module transforms complex temporal relationships into dynamic graph modeling, the core detection module identifies anomalies, and the back-end diagnostic module provides intuitive and interpretable analysis. This entire process ensures both detection accuracy and operability and reliability, providing a complete technical solution for log anomaly monitoring in complex distributed systems.

[0111] like Figure 2 As shown in the figure, an embodiment of the present invention provides a log anomaly detection method based on large language model-enhanced dynamic graph relation learning and interpretable diagnosis, comprising the following steps:

[0112] Step 1, LLM-enhanced log parsing and field extraction: Real-time log stream is pulled, parsing and cleaning are completed, and LLM is used to complete field extraction and semantic embedding generation;

[0113] Step 2, Construction of a multi-scale continuous-time dynamic graph: Insert nodes and edges based on event timestamps and field relationships;

[0114] Step 3, Graph Anomaly Detection: Run the GCN-Transformer encoder to calculate edge anomaly scores and mark suspicious relationships;

[0115] Step 4, Explainable Anomaly Diagnosis and Root Cause Analysis: Input the anomaly context into LLM to generate natural language explanations and root cause suggestions. Push the anomaly report and explanation to the operation and maintenance platform and support closed-loop optimization with human feedback.

[0116] In one implementation, the log parsing and field extraction module first preprocesses the raw log stream collected by the distributed system. This preprocessing includes timestamp alignment, noise reduction, and block segmentation to ensure the integrity and consistency of the log sequence. Based on this, a pre-trained large language model (e.g., BART or Qwen) is used to semantically parse the log content, dividing the log into templates and parameters. Key field information, including but not limited to IP addresses, error codes, and thread IDs, is extracted using named entity recognition technology. The parsing results are then converted into deep semantic vectors via an embedding layer for subsequent graph modeling.

[0117] In another implementation, the system uses a multi-scale continuous-time dynamic graph construction module to map log events and their field instances as graph nodes, and establishes graph edges based on the temporal order and interaction relationships between nodes. This dynamic graph supports zero-hop intra-event relationship modeling and can be extended to one-hop and multi-hop cross-event dependencies, thereby achieving unified modeling of local temporal sequences and global causal chains in log data. Through multi-scale modeling, the continuous evolution characteristics of log behavior can be fully represented.

[0118] In a further implementation, the enhanced graph representation learning and anomaly detection module receives the dynamic graph structure and jointly models the graph topology and temporal dependencies using a combination of Graph Convolutional Networks (GCNs) and Set-Transformers. To address the data imbalance problem caused by the scarcity of anomalous log samples, this module introduces a category-guided Mixup augmentation and contrastive learning strategy to improve the model's robustness under limited sample sizes and long-tailed distributions. Through the above training strategies, the system can output anomaly scores at the edge level or subgraph level, achieving accurate identification of log anomaly patterns.

[0119] In another implementation, the anomaly diagnosis and root cause analysis module receives the abnormal edges or subgraphs identified by the detection module, and organizes them into structured prompts based on contextual information. These prompts are then input into a large language model for reasoning. The large language model generates diagnostic results in natural language based on the prompt content, including explanations of the anomalies, inferences about potential root causes, and corresponding troubleshooting suggestions, thereby assisting maintenance personnel in quickly locating and resolving problems.

[0120] During overall operation, this embodiment of the invention sequentially connects log parsing, dynamic graph modeling, anomaly detection, and diagnostic interpretation to form a closed-loop mechanism of "detection-diagnosis-interpretation." This closed-loop mechanism not only ensures the accuracy and real-time performance of anomaly detection but also provides interpretable diagnostic output, enhancing the system's operability and reliability. Therefore, this embodiment of the invention provides a complete, reliable, and practically valuable technical solution for log anomaly monitoring in complex distributed systems.

[0121] The specific steps are as follows:

[0122] S1: LLM-enhanced log parsing and field extraction.

[0123] S11, use the Drain parser to process the raw logs of BGL to obtain the log template and parameters.

[0124] S12 uses the finely tuned Qwen1.5 model to extract fields from the parameters and identify key fields such as node ID and IP address.

[0125] S13, for each log template and field, use Qwen1.5 to generate its initial semantic embedding.

[0126] S14, Apply topic-aware semantic enhancement method to the log template: Use LDA to extract topic terms to enhance the log template L T The process involves, first, cleaning each log template obtained from the log parser, including removing special non-character tags and splitting camelCase or underscore-connected compound words into individual words. Each word in the preprocessed template is then processed. i The initial d-dimensional word vectors are generated using a lightweight word embedding model (such as Word2Vec). Simultaneously, topic-based LDA analysis is used to analyze the entire log template corpus, generating L-dimensional word vectors for each template. T Extract the k most representative keywords p k If the number of preprocessed template words is less than k, then all words are considered as topic words.

[0127] Finally, a topic-aware encoding formula is designed to calculate the w of each word. i In template LT The final vector v in i The formula takes into account both the word and all the keywords in the template. j The relative positional distance d of (j∈p, p is the set of keyword indices) ij (used to preserve word order information)

[0128] d ij =1 / (1+e) |i-j| ),

[0129] and semantic similarity Cos ij

[0130]

[0131] For non-topic terms w i Its final vector v i It is the result of its initial vector being weighted and adjusted by the two constraint factors mentioned above.

[0132]

[0133] Where m is the number of keywords, and λ is a hyperparameter that balances position and semantic weights.

[0134] Calculate the refined vector {v1, v2, ..., v} of each word in the template. t Regarding the keywords themselves, v i =v i 1. The final aggregation yields the final feature vector V of the template node. template .in

[0135] S5, the node feature vector V of each log event template. template And the node feature vector V of each extracted key field instance. field (Mainly derived from direct LLM embedding), together they constitute "Node Feature Vectors".

[0136] S2: Construction of multi-scale continuous-time dynamic graphs.

[0137] S21, use the log template and extracted key fields as graph nodes.

[0138] S22, construct a dynamic graph sequence with three scales: h=0, 1, and >1, based on the timestamps and relationships of log events. For example, the graph with h=1 connects log events that are temporally adjacent.

[0139] S23, according to the following formula

[0140] Calculate semantic similarity (SS): Calculate the connection between two nodes (v... i ,v j The cosine similarity between vectors is represented by .

[0141]

[0142] Calculate the Time Interval (TI): Calculate the difference in timestamps between two connected event nodes.

[0143] TI(v i ,v j ) = timestamp j -timestamp i ,

[0144] S3: Graph anomaly detection.

[0145] S31. Input the constructed multi-scale graph sequence into the GCN+Set Transformer encoder. Set the number of GCN layers to 3 and the hidden unit dimension to 768.

[0146] S32 uses a learning rate of 1e-3 and a weight decay of 5e-7 during training.

[0147] S33 applies category-guided Mixup to augment data on normal and anomalous samples at the hidden level.

[0148] S34, calculate the edge-level loss using the following formula:

[0149] L e =∑max{0,γ+f(pos)-f(neg)},

[0150] Where f(·) is the model's prediction function for outlier edges. f(pos) represents the model's predicted score for positive samples (normal edges), and f(neg) represents the predicted score for negative samples (outlier edges). γ is a preset interval. To improve training stability, selective negative sampling can be used, i.e., the loss is calculated only for sample pairs that the current model misclassifies (f(pos) ≤ f(neg)). The model's goal is to maximize the scores of normal edges and minimize the scores of outlier edges.

[0151] Calculate graph-level loss, graph-level loss (L g ): Introducing a graph-level objective, aiming to represent all normal graphs (obtained by pooling the hidden states of nodes, such as max pooling).

[0152]

[0153] Aggregate within a minimum hypersphere.

[0154]

[0155] Target L g is to learn to minimize the radius R of the hypersphere and the relaxation variable ε t ,

[0156]

[0157] where c and R are the center and radius of the hypersphere respectively, is the distance between the graph representation and the center, and ε t is for the relaxation variable introduced during training to accommodate outliers, and C is a hyperparameter used to balance the trade-off between the error ε t and the volume of the sphere.

[0158] Calculate the classification loss. For the pseudo-samples generated by Mixup and their mixed labels, calculate the standard cross-entropy loss,

[0159]

[0160] where y i ∈ {0, 1} is the label of sample i, and p i represents the predicted value of the template sequence, and M represents the number of sequences in each batch.

[0161] Calculate the contrastive loss,

[0162]

[0163] where M is a certain batch, the features of the normal log sequences are i ∈ [1, l], l < M, and l is the number of normal log sequences, and the features of the abnormal log sequences are j ∈ [1, M - l]. The features of the normal pseudo-samples after data augmentation are k ∈ [1, m], m < M, and m is the number of normal pseudo-log sequences, and the features of the abnormal pseudo-samples are g ∈ [1, M - m]. τ is the temperature coefficient, which is a positive hyperparameter used to adjust the discrimination of the model for negative samples. A smaller value of τ will make the model pay more attention to distinguishing those "difficult" negative samples that are similar to the positive samples, thus enhancing the learning effect.

[0164] Calculate the total loss,

[0165] L = L e + αL g + β(L cls + δLcon )+Regularization,

[0166] Where α, β, and δ are hyperparameters, and Regularization is the L2 regularization term for the model parameters. By minimizing this total loss, the model can simultaneously learn to distinguish between normal / abnormal edges, cluster normal graphs, and utilize data augmentation to improve generalization ability.

[0167] The aforementioned multi-granularity loss function is used for model optimization. This loss function integrates the objectives of edge-level, graph-level, classification, and contrastive learning.

[0168] S35 outputs the anomaly detection results.

[0169] S4: Explainable abnormality diagnosis and root cause analysis

[0170] S41, when the model detects an abnormal edge (e.g., a connection from node A to node B that should not exist), the system collects the original log text related to the edge, the features of nodes A and B, the features of the edge, and the abnormal score.

[0171] S42 integrates this information into a Prompt, for example: "You are a supercomputer operations expert. An abnormal correlation has been detected between node A (log event: 'kernel error') and node B (IP address: XXXX), with the time interval being abnormally shortened. Please analyze this anomaly and speculate on the possible causes."

[0172] S43. Input this prompt into the Qwen1.5 model. The model generates the following report:

[0173] Explanation of the anomaly: "The association between the detected 'kernel error' log and a specific IP address is extremely rare in normal mode, and the time interval between occurrences is much shorter than the historical average. This suggests that a sudden hardware or software failure may have occurred, rather than normal system operation."

[0174] Root cause recommendations: "1. Check the hardware health status of the compute node with IP address XXXX. 2. Review the detailed system logs related to kernel errors on this node. 3. Check for recent software updates or configuration changes."

[0175] To verify the effectiveness of the technical solution of this invention, we conducted detailed simulation experiments based on publicly available BGL, HDFS, and Thunderbird log datasets. The experiments compared the system of this invention with several mainstream baseline methods, and ablation experiments verified the necessity of each innovative module.

[0176] LLM-GLAD achieved excellent F1 scores on all datasets and demonstrated strong competitiveness in terms of precision and recall. This initially proves that LLM-GLAD can detect log anomalies more effectively than existing methods. Comparison results of different frameworks on the datasets are shown below. Figure 6 , Figure 7 , Figure 8 As shown.

[0177] Compared to the graph baseline: The performance improvement of LLM-GLAD compared to the original GLAD demonstrates the effectiveness of LLM augmentation, semantic and data augmentation modules, and multi-scale graph modeling. Compared to LogSD and TempoLog, LLM-GLAD exhibits unique advantages through deeper LLM integration (e.g., field extraction, node representation, and final interpretation) and specific attention to relational anomalies.

[0178] Compared to sequence-based and traditional methods, LLM-GLAD significantly outperforms traditional statistical methods and sequence-based deep learning methods. This highlights the importance of graph structures in capturing complex relationships between logs and the role of LLM in understanding log content, which are difficult for methods relying solely on sequence patterns or simple statistical features to achieve.

[0179] Performance of LLM-Only Baselines: While baselines using only LLM for sequence classification (whether zero-shot or fine-tuned) perform reasonably well, they still lag behind LLM-GLAD. This highlights the importance of graph structures and mechanisms specifically designed for relational anomalies in fully leveraging the capabilities of LLM. Simply applying LLM to log sequences may not fully capture the structured relationships between logs.

[0180] To demonstrate the indispensability of each innovative module in this invention, we designed ablation experiments. Based on the complete model, one key module was removed at a time, and the changes in its F1 score on the BGL, HDFS, and Thunderbird datasets were observed. The ablation experiment results of this invention on various datasets are as follows: Figure 9 As shown.

[0181] After removing the LLM-enhanced field extraction module, the performance dropped significantly, with the F1 score decreasing from 92.91 to 89.79, indicating that high-precision field extraction based on LLM is crucial for accurate subsequent relation modeling.

[0182] LLM embedding contribution: The node representation generated using LLM also brought performance improvement compared to using Sentence-BERT alone, with F1 increasing from 90.48 to 92.91, demonstrating the advantage of the more powerful LLM in capturing the deep semantics of nodes.

[0183] The effect of semantic enhancement: After removing the topic-aware semantic enhancement module, the performance decreased slightly, with the F1 score dropping from 92.91 to 90.13, which proves that this fine-grained semantic information supplementation is beneficial for distinguishing similar but different log templates.

[0184] Effectiveness of data augmentation: After removing data augmentation strategies such as Mixup and contrastive learning, the performance dropped significantly, with the F1 score decreasing from 92.91 to 85.07. This effect was particularly pronounced on datasets with sparser outlier samples, which validates their role in mitigating class imbalance and improving model robustness.

[0185] Advantages of multi-scale graphs: Replacing multi-scale graphs with single-scale graphs leads to a performance decrease, with F1 dropping from 92.91 to 86.85, demonstrating that considering dependencies at different time scales (H-hops) simultaneously is necessary for capturing diverse anomalous patterns.

[0186] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0187] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A log anomaly detection system based on enhanced dynamic graph and interpretable diagnosis, characterized in that, Comprising: a parsing and extraction module for preprocessing log streams of a distributed system and generating log templates and semantic embeddings of key fields through a large language model; a dynamic graph construction module for constructing multi-scale continuous-time dynamic graphs based on the temporal relationships of log events and field instances; a graph anomaly detection module for jointly modeling graph structure and temporal dependence through graph neural networks and attention encoders, outputting edge-level anomaly scores; a diagnosis and analysis module for organizing abnormal contexts into structured prompts and calling a large language model to generate interpretable abnormal diagnosis and root cause analysis results.

2. The system of claim 1, wherein, The parsing and extraction module uses a pre-trained language model to perform field-level named entity recognition on log templates and parameters, extracts key fields such as node identifiers, addresses, and task identifiers, and generates deep semantic embeddings.

3. A log anomaly detection method characterized by, Comprising the following steps: Step 1: Parsing and field extraction of real-time log streams, using a large language model to complete semantic embedding generation; Step 2: Constructing multi-scale continuous-time dynamic graphs based on event timestamps and field relationships; Step 3: Inputting the dynamic graph into a graph neural network and attention encoder combined model to calculate edge anomaly scores and label suspicious relationships; Step 4: Inputting abnormal contexts into a large language model to generate natural language explanations and root cause suggestions, and pushing them to the operation and maintenance platform.

4. The method of claim 3, wherein, The multi-scale dynamic graph in step 2 includes 0-hop, 1-hop, and greater-than-1-hop dependency modeling to capture short-range and long-range dependency features of log events.

5. A log parsing and feature enhancement method, characterized in that, Comprising: Drain parsing of original logs to obtain log templates and parameters; Using a fine-tuned language model to extract field information and generate semantic embeddings for log templates and fields; Using topic-aware encoding to enhance the semantics of log templates, adjusting word vectors through position distance weights and semantic similarity factors to obtain refined feature representations of template nodes.

6. The method of claim 5, wherein, The balance parameter in the topic-aware encoding ranges from 0.1 to 1, controlling the contribution proportion of position weight and semantic weight in the vector updating process.

7. A method of graph anomaly detection, the method comprising: Comprising: Inputting the constructed multi-scale dynamic graph into a joint model containing a 3-layer graph convolutional network and an attention encoder; Applying class-guided data mixing augmentation to hidden layers during training for normal and abnormal samples; Using a multi-granularity loss function that integrates edge-level, graph-level, classification, and contrastive learning for optimization and outputting anomaly scores.

8. The method of claim 7, wherein, The learning rate of the training is 0.001, and the weight decay coefficient is 0.0000005, and model parameter updating is achieved by minimizing the total loss. 9.A method for diagnosing log abnormality based on a large language model, characterized in that, Comprising: When an abnormal edge is detected, extract the context logs, related node features, and edge features of the edge; Organize the extracted information into structured prompts and input them into a large language model; Generating natural language diagnosis results containing abnormal explanations and root cause suggestions from the large language model.

10. The method of claim 9, wherein, The diagnosis results are pushed to the alarm system through the operation and maintenance platform interface, and manual feedback is supported to correct the model diagnosis prompt content.

Images