Log anomaly detection method and system based on directed attribute weighted graph neural network

Abstract

This application relates to the fields of artificial intelligence, computer system operation and maintenance, and log analysis, and discloses a method and system for log anomaly detection based on a directed attribute-weighted graph neural network. The method includes collecting a large amount of raw log messages and converting them into multiple structured log groups; converting each log group into a directed attribute-weighted graph containing semantic, temporal, and topological information; inputting the directed attribute-weighted graph into an attention-enhanced end-to-end single-class directed graph neural network, extracting the final representation of all nodes through attention-enhanced directed graph convolutional layers; aggregating the final representations of all nodes into a global representation vector of the directed attribute-weighted graph through graph representation learning; using the Euclidean distance between the global representation vector of the directed attribute-weighted graph and the center of the hypersphere as an anomaly score, and determining anomalies in the log groups corresponding to the directed attribute-weighted graph based on the anomaly score. This method can improve the detection accuracy of log anomalies.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence, computer system operation and maintenance and log analysis technology, and more specifically, to a log anomaly detection method and system based on a directed attribute-weighted graph neural network. Background Technology

[0002] With the rapid development of modern information technology, the scale of systems in fields such as cloud computing, big data, and intelligent manufacturing is constantly expanding, the number of components is growing exponentially, and system architectures are becoming increasingly complex. Traditional manual operation and maintenance methods can no longer meet the real-time monitoring needs of large-scale systems, making automated operation and maintenance an inevitable trend. System logs, as the core data carrier for recording system operating status and critical events, contain rich information about the system's operation. By analyzing log data, system anomalies can be detected in a timely manner, fault behaviors can be identified, and potential operational risks can be predicted. This is an important means of ensuring the stable, continuous, and efficient operation of the system.

[0003] Existing log anomaly detection methods can be mainly divided into three categories: quantitative statistical methods, such as Principal Component Analysis (PCA), One-Class Support Vector Machine (OCSVM), and Isolation Forest (iForest). These methods detect anomalies by constructing a log event counting matrix and analyzing anomalies in event frequency. However, these methods can only capture quantitative information about events, completely losing the semantic information of log events and the temporal relationships between events. They are extremely ineffective at detecting temporal and structural anomalies in complex systems.

[0004] Sequence modeling methods, such as DeepLog and LogAnomaly based on LSTM, treat logs as event sequences and use recurrent neural networks to capture the temporal dependencies between events. However, these methods can only capture local temporal relationships and cannot model complex topological structures and global dependencies between system components. When the system contains a large number of different types of log events, the model performance degrades sharply.

[0005] Graph-based methods: These methods convert logs into graph structures and use graph neural networks to capture the structural relationships between events. However, existing graph-based methods have the following serious defects: (1) Most of them use undirected graphs to represent the relationships between log events, completely losing the temporal sequence information of the events and failing to detect anomalies caused by the reversal of the event order; (2) Graph representation learning and anomaly detection processes are separated, making it impossible to optimize the graph representation for anomaly detection tasks, resulting in low detection accuracy; (3) They can only process graphs with no weight or simple weights, and cannot effectively utilize the frequency and time information of event transitions; (4) Most methods can only output anomaly detection results and cannot provide explainable anomaly causes, so maintenance personnel cannot quickly understand the cause of the anomaly and carry out maintenance and repair based on the detection results.

[0006] Furthermore, existing methods generally suffer from problems such as high requirements for training data purity, poor robustness, and inability to effectively identify structural anomalies. In real-world industrial scenarios, system log data is large in volume, complex in type, and scarce in anomaly samples. Moreover, system operating modes change dynamically over time, making it difficult for existing methods to meet the long-term stable operation and maintenance requirements. Summary of the Invention

[0007] To address the aforementioned issues, this invention provides a log anomaly detection method and system based on a directed attribute-weighted graph neural network, which can simultaneously capture the semantic, temporal, and topological information of log events, achieving high log anomaly detection accuracy and strong interpretability of the anomaly detection results.

[0008] To achieve the above objectives, according to a first aspect of the present invention, a log anomaly detection method based on a directed attribute-weighted graph neural network is provided. The method includes: collecting a large number of raw log messages and converting them into multiple structured log groups; converting each log group into a directed attribute-weighted graph containing semantic, temporal, and topological information; inputting the directed attribute-weighted graph into an attention-enhanced end-to-end single-class directed graph neural network, and extracting the final representation of all nodes through an attention-enhanced directed graph convolutional layer; aggregating the final representations of all nodes into a global representation vector of the directed attribute-weighted graph through graph representation learning; using the Euclidean distance between the global representation vector of the directed attribute-weighted graph and the center of the hypersphere as an anomaly score, and determining anomalies in the log groups corresponding to the directed attribute-weighted graph based on the anomaly score.

[0009] According to a second aspect of the present invention, a log anomaly detection system based on a directed attribute-weighted graph neural network is also provided. The system includes: a data preprocessing module configured to collect a large number of raw log messages and convert them into multiple structured log groups; a directed attribute-weighted graph construction module configured to convert each log group into a directed attribute-weighted graph containing semantic, temporal, and topological information; and an attention-enhanced end-to-end single-class directed graph neural network detection module configured to input the directed attribute-weighted graph into an attention-enhanced end-to-end single-class directed graph neural network, extract the final representation of all nodes through an attention-enhanced directed graph convolutional layer; aggregate the final representations of all nodes into a global representation vector of the directed attribute-weighted graph through graph representation learning; use the Euclidean distance between the global representation vector of the directed attribute-weighted graph and the center of the hypersphere as an anomaly score, and determine anomalies in the log groups corresponding to the directed attribute-weighted graph based on the anomaly score.

[0010] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: (1) The present invention provides a log anomaly detection method based on directed attribute weighted graph neural network. By converting the original log message into a directed attribute weighted graph, it can simultaneously capture the semantic, temporal and topological information of log events. It uses an attention-enhanced end-to-end single-class directed graph neural network to process each directed attribute weighted graph and tightly couples graph representation learning with the anomaly detection process, realizing end-to-end optimization and improving the accuracy of log anomaly detection. It is particularly suitable for real-time log monitoring and fault identification of complex distributed systems such as cloud computing platforms, intelligent manufacturing systems, and supercomputers.

[0011] (2) The log anomaly detection method based on directed attribute weighted graph neural network provided by the present invention can quantify the contribution of each node and edge to the anomaly score and generate an intuitive anomaly subgraph, clearly showing the core features and related log events that cause log group anomalies. The anomaly detection results are highly interpretable, providing accurate support for operation and maintenance personnel to carry out anomaly analysis and fault handling, and greatly improving operation and maintenance efficiency. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating a log anomaly detection method based on a directed attribute-weighted graph neural network provided in this application embodiment; Figure 2 A system architecture diagram of a log anomaly detection method based on a directed attribute-weighted graph neural network provided in this application embodiment; Figure 3 This is a flowchart illustrating the directed attribute weighted graph construction step in a log anomaly detection method based on a directed attribute weighted graph neural network, provided in an embodiment of this application. Figure 4 A schematic diagram of the structure of the attention-enhanced directed graph convolution Att-DiGCN layer in a log anomaly detection method based on a directed attribute-weighted graph neural network provided in this application embodiment; Figure 5 A flowchart illustrating the anomaly interpretation and analysis steps in an attention-enhanced end-to-end single-class directed graph neural network, provided as an embodiment of this application; Figure 6 This is an architecture diagram of the online deployment steps in an attention-enhanced end-to-end single-class directed graph neural network, provided in an embodiment of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0015] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0016] like Figure 1 As shown, a log anomaly detection method based on a directed attribute-weighted graph neural network is provided. This method is particularly suitable for real-time log monitoring and fault identification in complex distributed systems such as cloud computing platforms, intelligent manufacturing systems, and supercomputers, possessing strong industrial applicability and scenario adaptability. This method can be executed by a terminal or by a server communicating with the terminal via a network. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. The server can be a standalone server or a server cluster consisting of multiple servers. The method is illustrated using a terminal as an example, including the following steps: Step 101: Collect a large number of raw log messages and convert them into multiple structured log groups.

[0017] Step 102: Convert each log group into a directed attribute weighted graph containing semantic, temporal, and topological information.

[0018] Step 103: Input the directed attribute weighted graph into the attention-enhanced end-to-end single-class directed graph neural network, and extract the final representation of all nodes through the attention-enhanced directed graph convolutional layer.

[0019] Step 104: Through graph representation learning, the final representations of all nodes are aggregated into a global representation vector of a directed attribute-weighted graph.

[0020] Step 105: Use the Euclidean distance between the global representation vector of the directed attribute weighted graph and the center of the hypersphere as the anomaly score, and determine the anomaly of the log group corresponding to the directed attribute weighted graph based on the anomaly score.

[0021] In the above-mentioned log anomaly detection method based on directed attribute weighted graph neural network, the original log message is converted into multiple directed attribute weighted graphs. Each directed attribute weighted graph is processed simultaneously by an attention-enhanced end-to-end single-class directed graph neural network. The graph representation learning is tightly coupled with the anomaly detection process, achieving end-to-end optimization and thus improving the accuracy of log anomaly detection.

[0022] In one embodiment, such as Figure 2 As shown, a log anomaly detection system based on directed attribute weighted graph neural network is provided. The system converts the original log messages into directed attribute weighted graphs containing semantic, temporal and topological information, uses an end-to-end single-class directed graph neural network for anomaly detection, and combines hierarchical correlation propagation and gradient weighted activation mapping to achieve accurate explanation of the causes of anomalies. At the same time, the system has been optimized and deployed for the Huawei Atlas300 AI accelerator card, making it suitable for industrial-grade operation and maintenance scenarios.

[0023] The log anomaly detection system based on directed attribute weighted graph neural network includes the following core modules: (1) data preprocessing module, (2) directed attribute weighted graph construction module, (3) attention-enhanced end-to-end single-class directed graph neural network (Att-OC-DiGCN) detection module, (4) anomaly interpretation module, and (5) online deployment and incremental update module.

[0024] (1) Data preprocessing module The data preprocessing module is responsible for converting raw, unstructured log data into structured, standardized information, providing high-quality input for subsequent graph construction and model training, filtering out invalid noise data in the raw log messages, unifying data dimensions and formats, and ensuring the accuracy of subsequent model feature extraction and anomaly identification. The specific steps are as follows.

[0025] Log parsing: An improved fixed-depth tree log parser is employed, which extracts log event templates and parameter information from raw log messages through an adaptive depth adjustment mechanism and template clustering algorithm. The improved parser can automatically adjust the tree depth to adapt to log formats of varying complexity, solving the problems of low accuracy and poor adaptability of traditional fixed-depth tree parsers when processing complex log messages, and effectively improving the accuracy and versatility of log parsing in multiple scenarios.

[0026] Semantic Embedding Generation: A pre-trained BERT-based model (hereinafter referred to as the BERT model) is used to generate context-sensitive semantic embeddings for log events. First, the log event template is segmented and preprocessed to remove stop words and special characters. Then, the processed text is input into the BERT model, and the mean of the layer outputs is taken as the semantic vector of the log event. Compared with traditional static word embedding methods, the semantic vectors generated by the BERT model can capture the contextual semantic information of log events, significantly improving the accuracy of semantic representation and providing semantic support for subsequent refined anomaly detection.

[0027] Log grouping: Log events are grouped into log sessions based on session identifiers (such as block ID, container ID, node ID) in the log events. For log events without session identifiers, an adaptive sliding window is used for grouping. The window size is dynamically adjusted according to the average arrival rate of log events and the event interval to ensure that each log group contains the complete system operation process and to guarantee the integrity of log timing and business logic.

[0028] Data normalization: Z-score normalization is performed on the statistical characteristics of log events (such as frequency of occurrence and average interval time), mapping the feature values ​​to a standard normal distribution with a mean of 0 and a variance of 1, eliminating the dimensional differences between different features, improving the convergence speed and training stability of the model, and avoiding model training bias caused by differences in feature values.

[0029] (2) Directed Attribute Weighted Graph Construction Module The directed attribute weighted graph construction module is responsible for converting each log group into a directed attribute weighted graph containing semantic, temporal, and topological information. It fully preserves all key relationships between log events, avoids the information loss problem of traditional log feature extraction methods, maximizes the retention of effective features of system operation, and provides structured data support for high-precision anomaly detection.

[0030] like Figure 3 As shown, the specific steps are as follows: Node definition: Each node represents a unique log event, and the node set... , where n is the number of unique log events in the log group.

[0031] Node attribute construction: Attribute vector for each node It consists of two parts: (1) Semantic attributes: the semantic vector of log events generated by the BERT model, with a dimension of 768; (2) Statistical attributes: the frequency of occurrence of log events in the log group, the average interval time, the earliest occurrence time and the latest occurrence time, with a dimension of 4.

[0032] The final node attribute vector is obtained by concatenating semantic and statistical attributes, and then reduced to the specified node attribute vector dimension d (default 128, 300 for complex datasets) through a linear layer.

[0033] Directed edge and weight construction: For the event sequence in the log group, traverse each event in chronological order. If the event... Then the incident occurred immediately. Then add a node To the node Directed edge . edge weight matrix The element in the i-th row and j-th column represents the directed edge in the directed attribute-weighted graph. The weights are calculated using the time decay counting method, and the calculation formula is as follows:

[0034] in, For event transition, let t represent the event that occurs for the tth time. To the event The transfer; T represents the event transfer in the log group. Total number of times; For the indicator function, when the event transitions... equal hour The value is 1 if it is set to 1, and 0 otherwise. The time decay factor is calculated using the following formula:

[0035] in, This is the time decay factor (default is 0.01). This is the end time of the current log group. For event transfer The occurrence time. The time decay factor makes the transition of recent events contribute more to the edge weights, which is more in line with the temporal characteristics of system operation and effectively improves the accuracy of temporal anomaly identification.

[0036] Graph representation: The final constructed directed attribute-weighted graph is represented as follows ,in Let be a set of directed edges. This is a matrix of node attributes. This is the edge weight matrix. For real domain, The number of nodes in a directed attribute-weighted graph, and the node attribute matrix. The i-th line is denoted as , , Let d represent the attribute vector of the i-th node in the directed attribute weighted graph, and d be the attribute vector dimension of the i-th node (each node). In this application, the attribute vector dimension of all nodes is the same and is a preset given value.

[0037] (3) Attention-enhanced end-to-end single-class directed graph neural network (Att-OC-DiGCN) detection module This module is the core innovation of this invention. It proposes an attention-enhanced one-class directed graph convolutional network (Att-OC-DiGCN, also known as the Att-OC-DiGCN model), which can simultaneously process directed, attributed, and weighted graphs. It tightly couples graph representation learning with the anomaly detection process, achieving end-to-end optimization and completely solving the accuracy loss problem caused by the separation of feature extraction and anomaly detection tasks in traditional models. It is suitable for anomaly recognition scenarios of various complex system logs.

[0038] (3.1) Attention-enhanced Directed Graph Convolutional Network (Att-DiGCN) layer To effectively capture neighborhood information of different orders in directed graphs and adaptively adjust the importance of different neighborhood information, this invention improves traditional directed graph convolution by introducing an attention mechanism and proposing an attention-enhanced directed graph convolution layer. This layer can autonomously focus on key graph structural features, weaken interference from invalid features, and improve the accuracy of model feature extraction.

[0039] An attention-enhanced directed graph convolutional layer is used to extract features from the node attribute matrix of the input directed attribute-weighted graph, resulting in the final representation of all nodes in the directed attribute-weighted graph. .

[0040] like Figure 4 As shown, the attention-enhanced directed graph convolutional layer performs the following formula on the input node attribute matrix. Neighborhood convolution operation:

[0041] in, For the first The output of the order-neighbor convolution operation, And it is an integer. For real domain, The number of nodes in a directed attribute-weighted graph. To output feature dimensions, For a trainable parameter matrix, It is a first-order neighborhood aggregation matrix. for Neighborhood aggregation matrix of order, This is a matrix of node attributes. The adaptive attention coefficient is calculated using the following formula:

[0042] in, For the Sigmoid activation function, For trainable attention vectors, This is a vector concatenation operation; Node attribute matrix The i-th row represents the attribute vector of the i-th node in the directed attribute weighted graph.

[0043] First-order neighborhood aggregation matrix The calculation formula is:

[0044] in, It is a first-order transition probability matrix. The adjacency matrix of a directed attribute-weighted graph, its element in the i-th row and j-th column. Indicates from node To the node Directed edge The weight, i.e. . It is the identity matrix. To add a self-loop adjacency matrix, for The degree matrix; for The approximate diagonalized eigenvector matrix is ​​calculated based on Personalized PageRank (PPR).

[0045] Neighborhood aggregation matrix The calculation formula is:

[0046] in, for 1-order nearest neighbor matrix for The diagonalized weight matrix.

[0047] The formula for calculating the order nearest neighbor matrix is: in, This is the intersection operation for matrix elements.

[0048] (3.2) Graph representation learning To aggregate node-level representations into graph-level representations, this invention employs the Set2Set readout function (Set2SetReadout function, a type of readout function; a readout function in graph neural networks aggregates the feature vectors of all nodes into a fixed-length graph-level feature vector). This function can capture the global structural information of the graph and the interaction relationships between nodes, comprehensively preserving the topological, temporal, and semantic features of the log graph. The calculation formula for the Set2Set Readout function is as follows:

[0049] in, For the i-th node (i.e., node) in the directed attribute weighted graph The final representation of ) Let be the global representation vector of the graph.

[0050] Compared to traditional Readout functions such as sum, mean, and max, the Set2Set Readout function can better capture the complex structural information of the graph, effectively avoid the problem of global feature loss, and significantly improve the accuracy of anomaly detection.

[0051] (3.3) Improved Uniclass Classification Loss Function To achieve end-to-end anomaly detection, this invention combines graph representation learning with a single-class classification task and adopts an improved Deep Support Vector Data Description (DeepSVDD) loss function. This function adds a contrastive learning loss term to the Deep SVDD loss to enhance the representation consistency of normal samples, reduce the difference in feature distribution of normal samples, increase the feature distance between normal and abnormal samples, and improve the model's discriminative ability.

[0052] Deep SVDD is an end-to-end deep single-class anomaly detection algorithm that uses a neural network to map data to a feature space and learns a minimum volume hypersphere to surround normal samples, using the distance from the sample to the center of the sphere to determine anomalies.

[0053] In this invention, "end-to-end" refers to the joint training of graph representation learning and anomaly detection classification tasks within the same neural network architecture, with both tasks jointly optimized using the same total loss function. Unlike existing technologies that separate graph representation learning and anomaly detection into two independent stages for separate optimization, the end-to-end architecture of this invention allows the graph representation learning process to be directly optimized for the anomaly detection task, thereby significantly improving detection accuracy.

[0054] Specifically, the formula for calculating the total loss function is as follows: .

[0055] in, The loss function for Deep SVDD, For the loss function of contrastive learning, This is the balance coefficient (default 0.1).

[0056] The formula for calculating the loss function of Deep SVDD is: .

[0057] Where M is the number of directed attribute-weighted graphs in the training set. This represents a weighted graph with the m-th directed attribute. To enhance attention, directed graph convolutional layers ( The set of all trainable parameters, including the trainable parameter matrix. Trainable attention vectors And all other trainable parameters. The global representation vector of the weighted graph of the m-th directed attribute (hereinafter referred to as...) (global representation vector). As the center of the super ball, This is the weight decay coefficient. For the first The parameter matrix of the layer, For the Euclidean norm, It is the Frobenius norm.

[0058] Super Ball Center A dynamic update mechanism is used, updating every 10 epochs (training rounds) during training. The calculation formula is as follows: .

[0059] In this embodiment, by adopting a dynamic update mechanism, the hypersphere collapse problem can be effectively prevented, model training failure can be avoided, and the training stability and convergence effect of the model can be greatly improved.

[0060] The loss function for contrastive learning is Information Noise-Contrastive Estimation (InfoNCE) loss, calculated as follows:

[0061] in, for The global representation vector, for The representation vector of the enhanced view (generated by randomly deleting edges and nodes). The cosine similarity function is used. This represents the temperature coefficient (default is 0.1). Contrastive learning loss enhances the consistency of normal sample representations, strengthens the model's ability to learn normal operating modes, and significantly improves the model's ability to distinguish abnormal samples.

[0062] It should be noted that the definitions of the total loss function and its components described above are inherent characteristics of the attention-enhanced end-to-end single-class directed graph neural network architecture itself, and not merely temporary configurations used only during the training phase. This total loss function defines the direction of model parameter optimization; after training, the model directly calculates the anomaly score using the trained parameters.

[0063] (3.4) Calculation of outlier scores After training, for the input Its abnormal score is defined as The Euclidean distance between the global representation vector and the center of the hypersphere:

[0064] A higher anomaly score indicates a higher probability of anomalies in the corresponding log group. This can be addressed by setting an appropriate threshold. ,when When the log group is identified as abnormal, the automated anomaly identification process is completed.

[0065] The core advantage of this module lies in the fact that the training and inference phases use the exact same data input format and preprocessing procedures. During training, the input is a directed attribute-weighted graph constructed from historical normal logs, used to learn the representation space of normal patterns and determine the hypersphere center. During actual anomaly detection, the input is a directed attribute-weighted graph constructed from real-time collected logs, and the model directly outputs the distance from the global representation vector of this graph to the hypersphere center as the anomaly score. The graph construction rules, node attribute generation methods, edge weight calculation methods, and model forward propagation paths are completely consistent between the training and inference phases. There is no need to introduce additional feature extraction or preprocessing modules during the inference phase, and the trained model can be directly deployed for anomaly detection in real-time log streams.

[0066] (4) Exception Explanation Module To provide interpretable anomaly detection results and help operations and maintenance personnel quickly understand the causes of anomalies and grasp their key characteristics, this invention proposes an anomaly interpretation method that combines hierarchical correlation propagation (LRP) and gradient-weighted class activation mapping (Grad-CAM). This method can quantify the contribution of each node and edge to the anomaly score and generate an anomaly subgraph, which intuitively displays the core log events and event transition anomaly characteristics corresponding to the anomaly.

[0067] like Figure 5 As shown, the specific steps are as follows: (4.1) Node importance calculation: The importance score of the node is calculated using the improved leave-one-out method. The difference between the improved leave-one-out method and the traditional leave-one-out method is that gradient information is introduced to improve the calculation efficiency and judgment accuracy.

[0068] The formula for calculating the importance score of a node is:

[0069] in, For the j-th node in the directed attribute-weighted graph, For nodes The final expression, outlier scores For nodes The final expression gradient, To remove a node The abnormal scores afterwards.

[0070] The higher the importance score of a node, the greater the contribution of the log event corresponding to that node to the anomaly score, and the more likely it is to be the core related event that caused the log anomaly.

[0071] (4.2) Edge Importance Calculation: The importance score of an edge is determined by the gradient of the importance scores and outlier scores of the two nodes it connects with respect to the edge weight, as shown in the formula:

[0072] in, Edges with outlier scores The gradient of the weights.

[0073] The higher the importance score of an edge, the greater the contribution of the abnormal time-series transition relationship of the log events corresponding to that edge to the anomaly score, which is the core reason for log anomalies.

[0074] (4.3) Anomaly Subgraph Generation: Based on the importance scores of nodes and edges, select the top nodes with the highest importance scores. An anomaly subgraph is generated by connecting nodes and edges to these nodes. The anomaly subgraph visually displays the key events that cause the anomaly and the event transitions, providing clear visual support for operations and maintenance personnel to analyze anomalies and handle faults.

[0075] (4.4) Anomaly Feature Analysis: By analyzing the event types and event transfer patterns in the anomaly subgraph, combined with the system topology and business logic, the core features of the anomaly are automatically sorted out, the anomaly triggering rules are summarized, and a detailed anomaly analysis report is generated to assist maintenance personnel in quickly completing fault handling and system optimization.

[0076] (5) Online deployment and incremental update module This invention optimizes the entire deployment process for the Huawei Atlas300 AI accelerator card, achieving high throughput and low latency online log anomaly detection, supporting efficient incremental updates, adapting to long-term system operation mode changes, and ensuring the stability of model detection performance.

[0077] (5.1) Deployment environment configuration Operating System: Ubuntu 22.04 LTS, Python Version: 3.12, Deep Learning Framework: PyTorch 2.1.0, Huawei AI Acceleration Library: CANN 7.0.0, Log Collection Tool: Fluentd, Inference Service Framework: FastAPI.

[0078] (5.2) Model transformation and optimization To fully utilize the NPU acceleration capabilities of Huawei Atlas300, the trained PyTorch model was converted to Huawei's self-developed OM (Offline Model) format. Inference efficiency was improved through multi-dimensional operator optimization. The specific steps are as follows: ① Export the PyTorch model to ONNX format to standardize the model inference format; ② Use Huawei ATC tools to convert the ONNX model to OM format, while enabling optimization options such as operator fusion, constant folding, and quantization to simplify the model structure and reduce inference power consumption; ③ Adjust the model's input / output format and batch size according to the hardware characteristics of Atlas300 to maximize NPU utilization and achieve efficient use of computing resources.

[0079] (5.3) Online testing process The online testing process is fully automated, requiring no manual intervention, and is suitable for real-time industrial operation and maintenance scenarios. For example... Figure 6As shown, the specific process is as follows: ① Real-time log collection: Real-time log data is collected from various system nodes through Fluentd (an open-source unified log collector) to ensure the integrity and real-time nature of log data collection; ② Real-time preprocessing: The collected logs are parsed, semantically embedded, generated, and grouped to quickly complete the standardization of raw data; ③ Real-time graph construction: Each log group is converted into a directed attribute weighted graph to generate structured feature data that the model can recognize; ④ Model inference: The graph is input into the Att-OC-DiGCN model deployed on Atlas300 to quickly calculate the anomaly score; ⑤ Anomaly detection: The anomaly score is compared with a preset threshold to accurately determine whether there are anomalies in the log data; ⑥ Alarm triggering and response: If an anomaly is detected, an alarm is triggered immediately, and an anomaly explanation and feature analysis report are generated to promptly notify the operation and maintenance personnel for handling; ⑦ Data storage: Normal and abnormal log data are stored in the database for subsequent incremental model updates and operation and maintenance data analysis.

[0080] (5.4) Incremental update mechanism To adapt to dynamic changes in system operation modes and address the issues of rigidity and poor adaptability of traditional models, this invention employs an incremental learning mechanism to achieve online model updates without requiring system downtime for retraining, ensuring continuous operation and maintenance services. The specific steps are as follows: ① Regularly collect new normal log data to construct a new directed attribute weighted graph and accumulate the latest system operation feature data; ② Freeze the model's bottom convolutional layers, only fine-tuning the top Readout layer and single-class classification layer, adapting to new data features while retaining the original effective feature learning capabilities; ③ Fine-tune the model using new graph data, synchronously updating the hypersphere center to adapt to the latest normal system operation mode; ④ Evaluate the performance of the updated model. If the performance improvement reaches a preset threshold, replace the old model and continuously optimize detection accuracy.

[0081] The incremental update mechanism can quickly adapt to the new operating mode of the system without retraining the entire model, continuously optimize model performance, and ensure long-term stable and high-precision anomaly detection results.

[0082] Compared with the prior art, the system of the present invention has the following beneficial effects: (1) High detection accuracy: The log data is represented by a directed attribute weighted graph, which captures the semantic information, temporal order and topological dependencies between system components of the log events. The proposed attention-enhanced end-to-end single-class directed graph neural network tightly couples graph representation learning with the anomaly detection process and optimizes the representation space for the anomaly detection task. On multiple benchmark datasets, the ROC AUC is ≥0.9, PR AUC is ≥0.85, and the accuracy and recall are both ≥85%, which is significantly better than existing log anomaly detection methods.

[0083] (2) Strong interpretability: The proposed anomaly interpretation method combining LRP and Grad-CAM can quantify the contribution of each node and edge to the anomaly score and generate an intuitive anomaly subgraph, clearly displaying the core features of the anomaly and related log events, providing accurate support for operation and maintenance personnel to carry out anomaly analysis and fault handling, and greatly improving operation and maintenance efficiency.

[0084] (3) Excellent deployment performance: The deployment of Huawei Atlas300 AI accelerator card has been optimized throughout the entire process. A single Atlas300 card can support processing 1200+ logs per second with an inference latency of <80ms, meeting the real-time and high throughput requirements of industrial-grade systems.

[0085] (4) Good robustness: The Deep SVDD loss function with added contrastive learning term is adopted, which enhances the consistency of the model's representation of normal samples and improves the model's robustness to training data contamination; it supports incremental learning, can quickly adapt to the dynamic changes in the system's operating mode, and has strong long-term operating stability.

[0086] (5) Wide range of applications: This invention is applicable to log anomaly detection in various complex systems, including cloud computing platforms, distributed storage systems, supercomputers, intelligent manufacturing systems, etc. It can adapt to the system operation and maintenance needs of different architectures and different business scenarios, and has broad application prospects and practical value.

[0087] In one specific embodiment, taking the log data of a large distributed file system as an example, the specific implementation process of the present invention is described in detail, and the application effect of the present invention in the operation and maintenance scenario of a distributed storage system is verified.

[0088] The dataset used in this embodiment contains log data generated by 200 nodes running for 30 days, totaling approximately 570,000 log messages and including 48 different log event templates. Among them, there are approximately 16,000 abnormal log groups, with an anomaly rate of approximately 2.9%. The logs contain block IDs as session identifiers for log grouping. The dataset covers various operating scenarios such as normal system read / write, error reporting, and data retries, closely matching the actual industrial operation and maintenance environment.

[0089] Step S11, Data Preprocessing: (1) Log Parsing: The raw logs are parsed using an improved fixed-depth tree log parser to extract log event templates. The parser's maximum depth is set to 4, the minimum support is set to 10, and the parsing accuracy reaches 95%, which can accurately adapt to various formats of log data in the system.

[0090] (2) Semantic embedding generation: The semantic vector of the log event is generated using a pre-trained BERT-base model. First, the log event template is segmented to remove stop words and special characters. Then, the processed text is input into the BERT model, and the mean of the layer output is taken as the semantic vector with a dimension of 768, thus preserving the semantic features of the log event.

[0091] (3) Log grouping: Log messages are grouped into log groups according to block ID, resulting in approximately 570,000 log groups. Each log group corresponds to a complete file read / write business process.

[0092] (4) Data normalization: Z-score normalization is performed on statistical features such as the frequency of occurrence and average interval of log events to unify data dimensions and eliminate the impact of differences in scale on model training.

[0093] Step S12, Directed Attribute Weighted Graph Construction: Construct a directed attribute weighted graph for each log group to fully preserve business time-series, semantic, and topological features. Specific parameters are as follows: Nodes: Each unique log event corresponds to one node, with an average of 7 nodes per graph; Node attributes: Concatenate BERT semantic vectors (768 dimensions) and statistical features (4 dimensions), then reduce the dimensionality to 128 dimensions through a linear layer to balance feature integrity and model computation efficiency; Directed edges and weights: Traverse the log event sequence in chronological order and add directed edges. Edge weights are calculated using the time decay counting method, with a time decay coefficient λ=0.01. On average, each graph contains 20 edges, accurately restoring the event time-series transition pattern.

[0094] In the end, approximately 570,000 directed attribute weighted graphs were constructed, including about 16,000 anomaly graphs, which fully covered all log scenarios in the dataset and provided sufficient and high-quality sample data for model training.

[0095] Step S13, Model Training: (1) Dataset Partitioning: Divide the dataset into a training set (70% normal samples), a validation set (5% normal samples + 5% abnormal samples) and a test set (25% normal samples + 95% abnormal samples). The partitioning ratio is in line with the actual situation of the scarcity of abnormal samples in industrial scenarios.

[0096] (2) Model configuration: Att-DiGCN layer: 1 layer; convolution output dimension: 128; neighbor parameter k: 1; teleport probability α: 0.1; Readout function: Set2Set; optimizer: AdamW; learning rate: 0.001; weight decay coefficient λ: 0.0001; contrastive learning balance coefficient β: 0.1; batch size: 32; number of training rounds: 200.

[0097] (3) Training environment: Hardware: Huawei Atlas300I Pro inference card (32GB video memory); Software: Python 3.12, PyTorch 2.1.0, CANN 7.0.0.

[0098] (4) Training process: Initialize model parameters and hypersphere center o; input 32 graphs in each batch, calculate graph representation and loss function through forward propagation; update model parameters through backpropagation and iteratively optimize model weights; update hypersphere center o every 10 epochs (number of training rounds) to ensure model training stability; evaluate model performance in real time on the validation set, save the optimal model parameters, and avoid overfitting.

[0099] Step S14, Model Evaluation: A comprehensive performance evaluation of the trained model was conducted on the test set. All indicators met the industrial operation and maintenance standards. The specific results are as follows: ROC AUC: 0.912; PR AUC: 0.873; Accuracy: 0.885; Recall: 0.862; F1-score: 0.873.

[0100] The results show that the method of the present invention achieves excellent performance in distributed file system log anomaly detection tasks, with accuracy and recall rates far exceeding the industry standard of 80%. It can accurately identify various implicit and explicit log anomalies and is suitable for the operation and maintenance needs of distributed systems.

[0101] Step S15, Anomaly Interpretation and Analysis: For detected abnormal log groups, the system automatically calculates the importance score of each node and edge, generating a visual anomaly subgraph. For example, in an abnormal log group, the node "WriteBlock (WithException)" has the highest importance score, and the edge "ReceiveBlock→WriteBlock (WithException)" has a significantly higher importance score than other edges, intuitively reflecting the core associated data block write anomaly event and abnormal timing transition behavior. Operations personnel can rely on this anomaly analysis result to quickly locate abnormal system behavior and carry out targeted system troubleshooting and repair work.

[0102] Step S16, Online Deployment: Convert the trained model to OM format and deploy it on the Huawei Atlas300I Pro inference card. Build a FastAPI high-concurrency inference service to achieve 24 / 7 uninterrupted online detection. The core performance indicators of the online detection system are as follows: Log processing throughput: 1250 records / second; Single log group inference latency: 72ms; Alarm response time: <1 second.

[0103] The online system can collect, process, and analyze log data from the distributed file system in real time, promptly detect various operational anomalies and trigger alarms, and synchronously generate visual anomaly analysis reports, significantly reducing manual maintenance costs, improving system fault handling efficiency, and ensuring the continuous and stable operation of the distributed file system.

[0104] In another specific embodiment, taking the log data of a supercomputer system as an example, the specific implementation process of the present invention is described in detail, further verifying the detection effectiveness and scenario adaptability of the present invention in high-precision and complex systems such as supercomputing.

[0105] The dataset used in this embodiment contains log data generated by a supercomputer system running for one month, totaling approximately 69,000 log messages and including 1,848 different log event templates. Among them, there are approximately 31,000 abnormal log groups, with an anomaly rate of approximately 45.3%. The logs include node IDs as session identifiers for log grouping. The dataset covers log data from all scenarios, including supercomputing power scheduling, node operation, task execution, and error retries, making it far more complex than conventional operation and maintenance scenarios.

[0106] Step S21, Data Preprocessing and Graph Construction: Similar to Example 1, an improved fixed-depth tree log parser is used for log parsing. The BERT-base model generates semantic embeddings, logs are grouped according to node IDs, and a directed attribute-weighted graph is constructed. Due to the diverse log event types and high feature complexity of this dataset, the node attribute dimension is set to 300 to fully retain refined features. On average, each graph contains 10 nodes and 30 edges, completely reconstructing the complex operational topology and temporal logic of the supercomputing system.

[0107] Step S22, Model Training and Evaluation: The model configuration is basically the same as in Example 1, with some parameters fine-tuned based on the characteristics of supercomputing log data. The specific adjustments are as follows: Batch size: 128; Number of training rounds: 100; Node attribute dimension: 300.

[0108] The final evaluation results on the test set are as follows: ROC AUC: 0.942; PR AUC: 0.905; Precision: 0.913; Recall: 0.897; F1-score: 0.905.

[0109] The results show that the method of the present invention can still maintain extremely high detection accuracy and stability in supercomputing log scenarios with high complexity and multiple event types, which is significantly better than traditional sequence-based and statistical detection methods and has a strong ability to adapt to complex scenarios.

[0110] Step S23, Deployment and Maintenance Instructions: Step S231, Environment Setup Steps: (1) Install Ubuntu 22.04 LTS operating system to build a stable service running environment; (2) Install Huawei Atlas300 driver and CANN7.0.0 development environment; (3) Create Python 3.12 virtual environment, install PyTorch2.1.0, PyTorch Geometric2.4.0, FastAPI0.109.0 and other dependent libraries to ensure the normal operation of the algorithm and service; (4) Download the pre-trained BERT-base model for log semantic feature extraction; (5) Deploy Fluentd log collection tool, configure multi-node log collection rules to ensure real-time and complete collection of log data.

[0111] Step S232, Model Training and Conversion: (1) Clean and organize historical log data, complete preprocessing and graph structure construction, and generate model training sample set; (2) Run training script, iteratively train Att-OC-DiGCN model, and save the optimal training parameters; (3) Export the trained PyTorch model to ONNX format and complete model format standardization; (4) Use ATC tool to convert ONNX model to OM format, enable hardware optimization options such as operator fusion and quantization, and improve inference performance.

[0112] Step S233, Online Service Deployment: (1) Write FastAPI inference service code, load the optimized OM model, and build a high-concurrency inference interface; (2) Configure service port, concurrency and timeout parameters to adapt to large-scale log real-time inference scenarios; (3) Start the inference service, conduct interface connectivity and performance tests to ensure stable service operation; (4) Configure Fluentd forwarding rules to push real-time logs from each node to the inference service; (5) Deploy the service monitoring system to monitor the service operation status, throughput, latency and other core indicators in real time.

[0113] Step S234, Model Maintenance and Update: (1) Regularly collect new log data from the system, screen valid normal samples, and complete data standardization processing; (2) Perform incremental fine-tuning of the model once a month to adapt to the latest operating mode of the system and maintain detection accuracy; (3) Conduct full data training once a quarter to update the underlying parameters of the model and optimize feature extraction capabilities; (4) Build a model performance monitoring mechanism. When the model accuracy and recall rate drop by more than 5%, promptly start the full retraining process to ensure stable model performance.

[0114] In one embodiment, this application also provides a log anomaly detection system based on a directed attribute-weighted graph neural network, comprising: a data preprocessing module configured to collect a large number of raw log messages and convert them into multiple structured log groups; a directed attribute-weighted graph construction module configured to convert each log group into a directed attribute-weighted graph containing semantic, temporal, and topological information; and an attention-enhanced end-to-end single-class directed graph neural network detection module configured to input the directed attribute-weighted graph into an attention-enhanced end-to-end single-class directed graph neural network, extract the final representation of all nodes through an attention-enhanced directed graph convolutional layer; aggregate the final representation of all nodes into a global representation vector of the directed attribute-weighted graph through graph representation learning; use the Euclidean distance between the global representation vector of the directed attribute-weighted graph and the center of the hypersphere as an anomaly score, and determine anomalies in the log groups corresponding to the directed attribute-weighted graph based on the anomaly score.

[0115] The modules in the aforementioned log anomaly detection system based on directed attribute-weighted graph neural networks can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0116] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0117] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0118] The above description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0119] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0120] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A log anomaly detection method based on a directed attribute-weighted graph neural network, characterized in that, include: Collect a large number of raw log messages and convert them into multiple structured log groups; Each log group is converted into a directed attribute-weighted graph containing semantic, temporal, and topological information; The directed attribute weighted graph is input into an attention-enhanced end-to-end single-class directed graph neural network, and the final representation of all nodes is extracted through the attention-enhanced directed graph convolutional layer. Through graph representation learning, the final representations of all the nodes are aggregated into a global representation vector of a directed attribute-weighted graph; The Euclidean distance between the global representation vector of the directed attribute weighted graph and the center of the hypersphere is used as the anomaly score, and the log group corresponding to the directed attribute weighted graph is judged as anomaly based on the anomaly score.

2. The log anomaly detection method based on a directed attribute-weighted graph neural network as described in claim 1, characterized in that, The process of converting each log group into a directed attribute-weighted graph containing semantic, temporal, and topological information includes: Each unique log event in each log group is treated as a node. The attribute vector of each node is obtained by concatenating semantic attributes and statistical attributes. The semantic attributes are the multi-dimensional semantic vector of the unique log event, and the statistical attributes include the occurrence frequency, average interval, earliest occurrence time and latest occurrence time of the unique log event in its respective log group. Traverse each unique log event in the log group in chronological order. If a second event occurs immediately after the first event, add a directed edge from the node corresponding to the first event to the node corresponding to the second event. The first event and the second event are any two unique log events. The weight of the edge is calculated using the time decay counting method. Based on the node set, directed edge set, node attribute matrix, and edge weight matrix of the log group, a directed attribute weighted graph is output.

3. The log anomaly detection method based on a directed attribute-weighted graph neural network as described in claim 1, characterized in that, The extraction of the final representation of all nodes through the attention-enhanced directed graph convolutional layer includes: The attention-enhanced directed graph convolutional layer extracts features from the node attribute matrix of the input directed attribute weighted graph, obtaining the final representation of all nodes in the directed attribute weighted graph; wherein, the attention-enhanced directed graph convolutional layer performs the following formula on the input node attribute matrix. Neighborhood convolution operation: in, For the first The output of the order-neighbor convolution operation, And it is an integer. For real domain, The number of nodes in a directed attribute-weighted graph. To output feature dimensions, Let be a trainable parameter matrix, and d be the dimension of the attribute vector of each node in the directed attribute-weighted graph, with values ​​set to predefined values. It is a first-order neighborhood aggregation matrix. for Neighborhood aggregation matrix of order, This is a matrix of node attributes. The adaptive attention coefficient is calculated using the following formula: in, For the Sigmoid activation function, For trainable attention vectors, This is a vector concatenation operation; Node attribute matrix The i-th row represents the attribute vector of the i-th node in the directed attribute weighted graph.

4. The log anomaly detection method based on a directed attribute-weighted graph neural network as described in claim 3, characterized in that, The first-order neighborhood aggregation matrix is ​​calculated according to the following formula: in, It is a first-order transition probability matrix. The adjacency matrix of a directed attribute-weighted graph, its element in the i-th row and j-th column. This represents the weight of the directed edge from the i-th node to the j-th node. It is the identity matrix. To add a self-loop adjacency matrix, for The degree matrix; for The approximate diagonalized feature vector matrix is ​​obtained based on personalized PageRank calculation.

5. The log anomaly detection method based on a directed attribute-weighted graph neural network as described in claim 3, characterized in that, The The order neighborhood aggregation matrix is ​​calculated using the following formula: in, for 1-order nearest neighbor matrix for The diagonalized weight matrix; The formula for calculating the order nearest neighbor matrix is: in, This is the intersection operation for matrix elements.

6. The log anomaly detection method based on a directed attribute-weighted graph neural network as described in claim 1, characterized in that, The total loss function used by the attention-enhanced end-to-end single-class directed graph neural network is: in, The loss function is used to describe deep support vector data. For the loss function of contrastive learning, This is the balance coefficient; The formula for calculating the loss function for deep support vector data description is: Where M is the number of directed attribute-weighted graphs. This represents a weighted graph of the m-th directed attribute. For attention-enhanced directed graph convolutional layers, the set of all trainable parameters is defined. Let be the global representation vector of the weighted graph of the m-th directed attribute. As the center of the super ball, This is the weight decay coefficient. For the first The parameter matrix of the layer, For the Euclidean norm, It is the Frobenius norm; The loss function for contrastive learning is the same as that for information-noise contrastive estimation, and its calculation formula is as follows: in, Let be the global representation vector of the weighted graph of the m-th directed attribute. Let be the representation vector of the enhanced view of the m-th directed attribute-weighted graph, generated by randomly deleting edges and nodes. The cosine similarity function is used. This is the temperature coefficient.

7. The log anomaly detection method based on a directed attribute-weighted graph neural network as described in claim 6, characterized in that, The hypersphere center employs a dynamic update mechanism, updating every 10 training epochs during the training of the attention-enhanced end-to-end single-class directed graph neural network according to the following formula: 。 8. The log anomaly detection method based on a directed attribute-weighted graph neural network as described in claim 1, characterized in that, The step of aggregating the final representations of all nodes into a global representation vector of a directed attribute-weighted graph through graph representation learning includes: Graph representation learning uses the Set2Set read function to aggregate the final representations of all nodes into a global representation vector of the directed attribute-weighted graph according to the following formula: in, Let i be the final representation of the i-th node in the directed attribute-weighted graph. This is the global representation vector of a directed attribute-weighted graph. For real domain, The number of nodes in a directed attribute-weighted graph. This is the output feature dimension.

9. The log anomaly detection method based on a directed attribute-weighted graph neural network as described in claim 1, characterized in that, Also includes: Calculate the importance score of a node using the following formula: in, For the j-th node in the directed attribute-weighted graph, For nodes The final expression, For abnormal score pairs of nodes The gradient is the final representation of the gradient. To remove a node The higher the importance score of a node, the greater the contribution of the log events corresponding to that node to the anomaly score. The importance score of an edge is determined by the gradient of the importance score and the anomaly score of the node with respect to the edge weight. The higher the importance score of an edge, the greater the contribution of the anomaly in the time sequence transition relationship of the log event corresponding to the edge to the anomaly score. Based on the importance scores of nodes and edges, select the top few nodes with the highest importance scores and the edges connecting these nodes to generate an abnormal subgraph.

10. A log anomaly detection system based on a directed attribute-weighted graph neural network, characterized in that, include: The data preprocessing module is configured to collect a large number of raw log messages and convert them into multiple structured log groups; The directed attribute weighted graph building module is configured to convert each log group into a directed attribute weighted graph containing semantic, temporal, and topological information; An attention-enhanced end-to-end single-class directed graph neural network detection module is configured to input the directed attribute-weighted graph into an attention-enhanced end-to-end single-class directed graph neural network, extract the final representation of all nodes through an attention-enhanced directed graph convolutional layer, aggregate the final representation of all nodes into a global representation vector of the directed attribute-weighted graph through graph representation learning, use the Euclidean distance between the global representation vector of the directed attribute-weighted graph and the center of the hypersphere as an anomaly score, and determine anomalies in the log groups corresponding to the directed attribute-weighted graph based on the anomaly score.

Images