GPU (Graphics Processing Unit) server fault positioning method based on retrieval enhancement generation and knowledge graph

Abstract

The invention provides a GPU operation and maintenance intelligent system integrating anomaly perception, semantic-level reasoning, causal chain reconstruction, intelligent repair decision and self-feedback learning. According to the system, interpretable root cause tracking is achieved through a retrieval-enhanced generation (RAG)-map hybrid reasoning mechanism, automatic repair action planning is achieved through a strategic generation model, and knowledge edge weight self-updating and reasoning ability self-evolution are achieved through verification feedback. According to the method, the normal form transition of GPU fault handling from post analysis to real-time deduction and self-healing regulation is realized, and a full-process intelligent closed loop of sustainable evolution is formed.

Description

Technical Field

[0001] This invention belongs to the field of intelligent server operation and maintenance technology, specifically relating to a GPU server fault location method based on retrieval-enhanced generation and knowledge graph. This method can be used in intelligent fault diagnosis systems for data center GPU server clusters, AI computing clusters, cloud platform GPU services, and large model training platforms. Background Technology

[0002] With the widespread adoption of large-scale model training, inference, and HPC (high-performance computing), the number of GPU servers in data centers is growing exponentially. GPU servers involve multiple layers of components, including graphics card hardware, drivers, CUDA runtime, container runtime, middleware, distributed training frameworks, and task scheduling systems. Their operating environment is highly complex, and the types of failures are diverse and often hidden.

[0003] Traditional GPU fault localization relies heavily on human experience. The process typically includes: monitoring system alarms, manually logging into the node to view logs; comparing with historical experience to determine possible root causes; manually executing verification scripts; and determining the final root cause and developing a repair plan based on the verification results. This method often results in scattered and unstructured knowledge; log formats are complex and semantically inconsistent; and most existing systems can only provide preset templates, failing to automatically generate personalized repair plans based on GPU model, task type, driver version, etc.

[0004] Therefore, there is an urgent need for a GPU fault location method that combines structured knowledge from knowledge graphs, intelligent retrieval-Augmented Generation (RAG) and LLM inference to achieve intelligent analysis across the entire chain. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes an intelligent fault location method for GPU servers based on the fusion of knowledge graphs and large language models. On the basis of deep learning and knowledge engineering theory, and taking into account the automation and intelligence requirements of GPU fault diagnosis, this invention incorporates multi-source log semantic fusion and causal reasoning mechanisms as core technologies. The improved RAG algorithm is used to improve the accuracy of fault root cause location, thereby improving the efficiency and reliability of fault diagnosis and significantly reducing the operation and maintenance costs and fault recovery time of GPU clusters.

[0006] The intelligent fault location method for GPU servers based on the fusion of knowledge graphs and large language models includes the following steps:

[0007] Step 1: Based on historical work orders, monitoring systems, driver logs, hardware information, and verification script results, and considering the completeness of the fault causal chain, construct a GPU fault knowledge graph based on an entity-relationship model. Entity types include Device, Event, Symptom, Root Cause, Action, and Validation. Relationship types include Event→Symptom, Symptom→Root Cause, Root Cause→Action, and Action→Validation. Store the graph in the Neo4j graph database.

[0008] Step 2: Use natural language processing technology to parse GPU driver logs, system logs, container logs, and deep learning framework logs into unified structured log events. Use Drain and Spell algorithms to classify log templates, construct log semantic vectors based on LogBERT or a self-trained Transformer model, and use One-Class SVM or AutoEncoder for log anomaly detection. Store the processed log vectors in Milvus or Elasticsearch vector databases. When an abnormal log event is detected, calculate the semantic similarity between the log and historical fault logs.

[0009] Step 3: Based on RAG technology, construct a two-stage reasoning process for fault diagnosis. The retrieval unit searches for nodes and edges that match the current symptoms from the knowledge graph and retrieves similar historical events from the log semantic vector library. It performs context filtering based on GPU model, driver version, and runtime environment. The generator generates a structured fault diagnosis report based on a large language model, including fault cause inference, fault propagation path, step-by-step repair plan, verification test plan, risk warning, and automated repair script template.

[0010] Step 4: Construct a context enhancement module, inject multi-dimensional meta-information such as GPU hardware features, software environment, running tasks, network topology, and monitoring metrics, encode the context information into feature vectors, and fuse them with log semantic vectors.

[0011] Step 5: Repeat steps 3 and 4, continuously updating the parameters of the knowledge graph and retrieval model based on work order verification feedback. Automatically update the graph edge weights based on the repair success rate, extract entities from newly added fault cases and expand the graph, incrementally fine-tune the retrieval model, and adopt the RLHF optimization generation strategy. When the training iteration count reaches the preset number of rounds or the model converges, save the optimized model. Use the saved model for real-time GPU fault diagnosis, and guide maintenance personnel or automated systems to perform repair operations based on the repair solutions output by the model, maximizing maintenance efficiency while ensuring diagnostic accuracy.

[0012] The present invention has the following beneficial effects:

[0013] 1. A novel knowledge graph construction method is proposed, which explicitly models the causal chain of GPU failures as a graph structure, supporting multi-hop reasoning and path tracing. This effectively solves the problem that traditional rule engines have difficulty expressing complex causal relationships.

[0014] 2. A multi-source log semantic fusion technology is proposed. This technology uniformly parses heterogeneous logs from drivers, systems, containers, frameworks, etc., and realizes cross-log source correlation analysis through semantic vectorization. This effectively solves the problem that traditional log analysis tools cannot correlate multi-source information.

[0015] 3. Intelligent reasoning is achieved by leveraging retrieval-enhanced generative techniques. This combines knowledge graph retrieval with large language model generation, and adaptively adjusts based on contextual features, effectively improving diagnostic accuracy and enhancing reasoning interpretability. A self-learning optimization mechanism is incorporated to achieve knowledge evolution, thereby enhancing the system's long-term availability. When applied to fault diagnosis of large-scale GPU clusters, it effectively addresses hardware diversity and environmental complexity. Attached Figure Description

[0016] Figure 1 A schematic diagram of a reinforcement learning model based on augmented exploration backtracking pruning;

[0017] Figure 2 This example shows how the model reward value changes with training rounds. Detailed Implementation

[0018] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0019] In the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a technical solution that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to the solution.

[0020] This invention provides a system architecture for an intelligent fault location method for GPU servers based on the fusion of knowledge graphs and large language models. The system architecture mainly includes the following functional modules: a log collection and normalization module, used to collect GPU driver logs, system logs, container logs, and monitoring metrics from multi-source heterogeneous systems, and perform structured parsing and semantic vectorization; a fault knowledge graph construction module, used to extract entities and relationships from historical work orders and logs to construct a GPU fault knowledge graph containing causal chains; a RAG intelligent reasoning module, used to automatically locate the root cause of faults and generate repair solutions based on retrieval-enhanced generation technology; a work order closed-loop management module, used to automatically generate work orders, execute repair actions, call verification scripts, and achieve closed-loop management; and a self-learning optimization module, used to automatically update the knowledge graph based on verification feedback, fine-tune the retrieval model, and achieve continuous evolution of the knowledge system.

[0021] The implementation process of this invention includes the following steps:

[0022] Step S11: The log collection and normalization module is used to collect multi-source heterogeneous logs from the GPU monitoring system, container platform, distributed training framework and system kernel, and parse them into unified structured log events through natural language processing technology, so as to realize semantic fusion and vectorized storage of multi-source data.

[0023] When an abnormal event occurs during the operation of the GPU server or the monitoring system detects abnormal performance indicators, the log collection module automatically collects log information from the following data sources: GPU driver logs, including nvidia-smi output, DCGM monitoring data, NVML interface logs, and Xid error code logs; system logs, including kernel logs, systemd service logs, dmesg output, and PCIe error logs; container platform logs, including Docker container logs, Containerd runtime logs, Kubelet logs, and Kubernetes Pod event logs; distributed training framework logs, including NCCL communication logs, PyTorch training logs, TensorFlow execution logs, and Horovod distributed logs; and monitoring indicator data, including time-series data such as GPU utilization, temperature, power consumption, frequency, voltage, and ECC error count.

[0024] When collecting logs, the system uses Filebeat or Fluentd as the log collection agent, transmits data through a Kafka message queue, and uses Schema Registry to unify the log structure. The log collection module preprocesses the received raw logs, including timestamp normalization, field extraction, and abnormal character filtering.

[0025] Specifically, the system uses BPE segmentation technology to preprocess the log text, and employs Drain or Spell algorithms to classify log templates, merging similar log events into a unified template. Subsequently, the system constructs log semantic vectors based on LogBERT or a self-trained Transformer model, mapping each log entry to a vector representation in a high-dimensional semantic space. The system then uses a One-Class SVM or AutoEncoder model to perform anomaly detection on the logs, identifying abnormal events that deviate from normal patterns.

[0026] The processed structured log data is written to Elasticsearch for full-text search, the log semantic vectors are written to the Milvus vector database for similarity retrieval, and the log associations are written to the Neo4j graph database for causal chain analysis.

[0027] Through the above settings, unified collection, structured parsing, and semantic vectorization of multi-source heterogeneous logs can be achieved, thereby enabling cross-system log correlation analysis in complex GPU cluster environments, avoiding information silos, and improving fault location efficiency.

[0028] Step S21: The fault knowledge graph construction module is used to extract entities and relationships from historical work orders, verification results, log events and expert annotations, and construct a GPU fault knowledge graph containing causal chains to realize the structured expression and visual management of fault knowledge.

[0029] After log collection is completed, the knowledge graph construction module extracts fault-related entities from multi-source data and establishes causal relationships between entities.

[0030] Specifically, the system defines the following entity types: Device entities include device attributes such as GPU model, driver version, MIG configuration, and PCIe topology; Event entities include triggering events such as system events, driver events, and framework events; Symptom entities include fault symptoms such as performance degradation, training interruption, memory anomalies, and communication timeouts; RootCause entities include root cause types such as hardware failure, driver conflict, configuration error, and resource contention; Action entities include repair actions such as restarting services, updating drivers, replacing hardware, and adjusting configurations; and Validation entities include validation methods such as diagnostic scripts, monitoring metrics, and functional tests.

[0031] The system defines the following relationship types: Event→Symptom indicates that an event causes symptoms, Symptom→RootCause indicates that symptoms point to the root cause, RootCause→Action indicates that the root cause is associated with a repair solution, and Action→Validation indicates that the repair action corresponds to the validation method.

[0032] The module employs a combination of rule extraction, statistical association, and graph neural networks for relationship mining. Rule extraction is based on expert knowledge to define causal rule templates; statistical association uses PMI (Point Mutual Information) and TF-IDF algorithms to calculate entity co-occurrence frequencies; and the graph neural network learns implicit entity relationships through a GNN model.

[0033] The completed knowledge graph is stored in the Neo4j graph database, supporting multi-hop path retrieval and causal chain tracing using the Cypher query language.

[0034] For example, when the "Xid 95 error" event is detected, the system automatically retrieves the causal chain in the knowledge graph: "Xid95 → ECC memory error → GPU memory degradation → reload driver + memtest", and extracts the corresponding repair actions and verification methods.

[0035] In addition, the knowledge graph construction module provides a visual management interface, allowing operations and maintenance personnel to view fault causal chains, edit entity relationships, and label new fault types in a graphical way, thus achieving knowledge management that combines manual and automated methods.

[0036] Step S22: In this embodiment, the RAG intelligent reasoning module is used to perform GPU fault analysis based on retrieval enhancement generation technology when the work order is in the "fault location" state. Specifically, the retrieval module first retrieves nodes and edges matching the current fault symptoms from the knowledge graph, and uses a graph traversal algorithm to extract relevant causal chain paths. The system retrieves historical fault events with semantic similarity to the current abnormal log from the log semantic vector library.

[0037] The retrieval module filters and sorts search results based on contextual information such as GPU model, driver version, CUDA version, running task type, and network topology, ensuring that the search results are highly relevant to the current environment. The system encodes the contextual information into a feature vector c, which is then fused with the log semantic vector v: root cause analysis and repair scheme generation are performed, achieving automated and intelligent fault diagnosis.

[0038] This module adopts a two-stage reasoning architecture, including two sub-modules: a retrieval unit and a generator unit. It achieves intelligent reasoning by combining knowledge graph retrieval with large language model generation.

[0039] The generator module uses a large language model to reason and generate retrieved knowledge. The system constructs a Prompt from the retrieved causal chains, historical cases, and contextual information, then calls an LLM (Language Modeling) to generate a structured fault diagnosis report. The generated content includes: fault cause inference, describing the root cause of the fault; fault propagation path, showing the complete link from symptoms to root cause; step-by-step repair plan, providing executable repair steps; verification and testing plan, specifying the verification script and detection indicators after repair; risk warnings and alerts, explaining potential risks during the repair process; and automated repair script templates, generating directly executable Shell or Python scripts.

[0040] The system will push the generated diagnostic report to the work order management system and automatically determine whether to trigger the automated repair process based on the feasibility of the repair plan.

[0041] With the above settings, after completing the fault retrieval, the system can generate high-quality diagnostic reports and repair solutions based on the reasoning capabilities of the large language model, achieving a deep integration of "knowledge retrieval" and "intelligent generation", and significantly improving the accuracy and interpretability of fault diagnosis.

[0042] Step S31: The work order closed-loop management module is used to automatically generate fault work orders based on RAG inference results, and execute repair actions, call verification scripts, and realize closed-loop management through the automated operation and maintenance platform to ensure the integrity and traceability of fault repair.

[0043] After the RAG module generates a diagnostic report, the work order management module automatically creates a fault work order and associates the diagnostic report, repair plan, verification script and other information with the work order.

[0044] Step S32: The work order management module determines the execution path based on the type of repair solution. For faults that can be automatically repaired at the software level, the system automatically calls the Ansible or Temporal automated operation and maintenance platform to execute the repair script. Specifically, the system sends the repair command to the target GPU node via SSH or Agent, and monitors the task execution status, output logs, and return codes in real time.

[0045] Step S33: After the repair task is completed, the work order management module automatically triggers the verification process. The verification execution module selects a target verification script from a preset verification script library based on metadata such as GPU model, driver version, and fault type. The script library includes: a GPU hardware detection script to detect GPU chip status, ECC errors, temperature, voltage, and utilization; a driver version comparison script to verify whether the driver, firmware, CUDA, and cuDNN versions match; a GPU runtime environment detection script to detect whether container services, runtime environment, and system processes are normal; a deep learning framework compatibility detection script to verify whether frameworks such as PyTorch and TensorFlow can load the GPU normally; and a performance benchmark test script to run benchmark tests such as matrix multiplication and convolution to verify whether performance has recovered.

[0046] Step S34: The verification execution module remotely executes the verification script through the automated operation and maintenance platform, and collects the execution results, output logs, performance indicators, and return codes in real time. The system compares the returned result fields, error codes, performance indicators, and expected thresholds with the script's parameters. If all key detection points meet expectations, the verification is successful, the system automatically updates the work order status to "closed," and archives the verification results, log files, and performance test results to the operation and maintenance log system. If any key detection point does not meet expectations, the verification fails, the system generates a structured anomaly report, marks the failure type, and automatically triggers the work order rollback process, rolling back the work order status to "fault location" or "pending repair," and restarting the diagnosis and repair process.

[0047] Step S35: After the verification process is completed, regardless of whether the verification is successful or not, the work order management module will archive the following data in a structured manner: work order identifier and creation time, fault type and root cause analysis, repair plan and execution command, verification script identifier and version, execution time and duration, verification results and output logs, hardware and software indicator snapshots, verification failure type and error code, and rollback trigger reason. The archived data is written to the operation and maintenance log system and knowledge graph database for subsequent fault tracing, model training, and knowledge evolution.

[0048] Step S41: The self-learning optimization module is used to automatically update the edge weights of the knowledge graph, expand and add new entities, fine-tune the retrieval model, and optimize the generation strategy based on the work order verification feedback, so as to realize the continuous evolution of the knowledge system and the automatic improvement of system performance.

[0049] After the work order is closed, the self-learning module extracts information such as verification results, repair success rate, and fault type from the archived data, and optimizes knowledge accordingly.

[0050] Step S42: The self-learning module automatically updates the edge weights in the knowledge graph based on the repair success rate. For successfully verified repair paths, the system increases the weight of the corresponding edge; for failed verification paths, the system decreases the weight of the corresponding edge.

[0051] Step S43: When the system detects a new fault type or repair solution, the self-learning module automatically extracts the new entity from the work order data and adds it to the knowledge graph. Specifically, the system uses a named entity recognition model to extract entities from fault descriptions and repair logs, and uses an entity linking algorithm to determine whether it is a new entity. If it is a new entity, it automatically creates a node and establishes a relationship with existing nodes.

[0052] Step S44: The self-learning module incrementally fine-tunes the retrieval model. The system uses newly added fault cases as training samples and updates the log semantic vector model using contrastive learning or supervised fine-tuning to improve the model's ability to identify new fault types. The fine-tuning process uses mini-batch gradient descent to avoid catastrophic forgetting.

[0053] Step S45: The self-learning module uses RLHF (Human Feedback Reinforcement Learning) technology to optimize the generation strategy. Based on the feedback scores from operations and maintenance personnel on the diagnostic reports, the system constructs a reward model and optimizes the LLM generation strategy through reinforcement learning algorithms, making the generated repair solutions more aligned with actual operations and maintenance needs.

[0054] Step S46: The self-learning module automatically archives newly added fault samples into the sample library and periodically triggers model retraining tasks. The system uses a scheduled task or sample quantity threshold triggering mechanism to ensure continuous model updates.

[0055] Through the above settings, the system achieves automated evolution of knowledge graphs, retrieval models, and generation strategies, thereby continuously improving the accuracy of fault diagnosis and the success rate of repair during long-term operation and maintenance, and significantly reducing the cost of manual intervention.

[0056] Step S51: During system operation, the log tracking module continuously collects all key events throughout the work order lifecycle in an event-driven manner, including work order creation and status change events, RAG inference process and result events, repair task execution start and end events, verification script call and return events, knowledge graph update events, and model fine-tuning trigger events. The module receives the above event data through Kafka and generates standardized structured log entries to provide basic data for subsequent analysis and visualization.

[0057] Step S52: The log tracking module performs structured processing on the collected event data, parsing and reassembling the text information of unstructured logs according to predefined fields. The log format includes: timestamp, execution module identifier, work order identifier, task node identifier, status value, execution result, error type and error code, event source, execution time, and additional contextual information. The module stores the data in JSON format and can interface with the Grafana monitoring system to achieve real-time visualization and historical traceability of the work order execution process.

[0058] Step S53: The log tracking module provides a multi-dimensional query interface, supporting log retrieval and analysis based on conditions such as work order identifier, fault type, GPU node, and time range. The system provides statistical analysis functions, including fault type distribution, repair success rate trends, average repair time, and verification failure cause analysis, providing data support for operation and maintenance decisions.

Claims

1. A method for intelligent fault localization of GPU servers based on the fusion of knowledge graph and large language model, comprising constructing a GPU fault knowledge graph, using Markov decision processes to describe the fault diagnosis and repair process, building a retrieval and generator neural network, and initializing network parameters, characterized in that: It also includes the following steps: Step 1: Collect multi-source heterogeneous logs from GPU monitoring systems, container platforms, distributed training frameworks, and system kernels, parse them into unified structured log events using natural language processing technology, and store them in semantic vector format. Step 2: Extract entities and relationships from historical work orders, verification results, log events and expert annotations to construct a GPU fault knowledge graph containing causal chains. The entities include device entities, event entities, symptom entities, root cause entities, repair action entities and verification entities. The relationships include events causing symptoms, symptoms pointing to root causes, root causes associated with repair solutions, and repair actions corresponding to verification methods. Step 3: Based on the retrieval enhancement generation technology, the automatic location of the root cause of the fault and the generation of the repair plan are realized. The retrieval module retrieves nodes and edges that match the current fault symptoms from the knowledge graph and retrieves historical fault events with similar semantics from the log semantic vector library. The generator module reasons and generates the retrieved knowledge based on the large language model and outputs a structured fault diagnosis report. Step 4: Automatically generate fault work orders based on the Retrieval-Augmented Generation (RAG) inference results, execute repair actions through the automated operation and maintenance platform, call verification scripts and achieve closed-loop management. If verification fails, the work order rollback process will be automatically triggered. Step 5: Based on the verification feedback, automatically update the edge weights of the knowledge graph, expand and add new entities, fine-tune the retrieval model, and optimize the generation strategy to achieve the continuous evolution of the knowledge system.

2. The intelligent fault location method for GPU servers based on the fusion of knowledge graph and large language model as described in claim 1, characterized in that: The logs include GPU driver logs, system logs, container platform logs, distributed training framework logs, and monitoring metric data. A log collection agent is used to transmit data through a message queue. The raw logs are standardized with timestamps, fields are extracted, and abnormal characters are filtered. A log template classification algorithm is used to merge similar log events into a unified template. Log semantic vectors are constructed based on a deep learning model. The processed structured log data is written to a search engine for full-text retrieval. The log semantic vectors are written to a vector database for similarity retrieval. The log associations are written to a graph database for causal chain analysis.

3. The intelligent fault location method for GPU servers based on the fusion of knowledge graph and large language model as described in claim 1, characterized in that: Relationship mining is performed by combining rule extraction, statistical association, and graph neural networks. The edge weights of the graph are dynamically weighted based on historical verification results. Paths that are successfully verified have their weights increased, while paths that are not verified have their weights decreased. The completed knowledge graph is stored in a graph database, supporting multi-hop path retrieval and causal chain tracing.

4. The intelligent fault location method for GPU servers based on the fusion of knowledge graph and large language model as described in claim 1, characterized in that: The retrieval module filters and sorts the retrieval results based on contextual information such as GPU model, driver version, and running task type. The generator module takes causal chains, historical cases, and contextual information as input and generates content including fault cause inference, fault propagation path, step-by-step repair plan, verification test plan, risk warning and alert, and automated repair script template. The output format adopts JSON structure.

5. The intelligent fault location method for GPU servers based on the fusion of knowledge graph and large language model as described in claim 1, characterized in that: For software-level faults that can be automatically repaired, the system automatically calls the automated operation and maintenance platform to execute the repair script. After the repair task is completed, the verification process is automatically triggered. The target verification script is selected from the verification script library according to the fault type. If the verification is successful, the work order status is updated to "closed" and the verification result is archived. If the verification fails, an exception report is generated and the work order rollback process is automatically triggered.

6. The intelligent fault location method for GPU servers based on the fusion of knowledge graph and large language model as described in claim 5, characterized in that: The verification script library includes GPU hardware detection scripts, driver version comparison scripts, GPU runtime environment detection scripts, deep learning framework compatibility detection scripts, and performance benchmark test scripts.

7. The intelligent fault location method for GPU servers based on the fusion of knowledge graph and large language model as described in claim 1, characterized in that: When the system detects a new fault type or repair solution, it automatically extracts new entities from the work order data, creates nodes to establish relationships, incrementally fine-tunes the retrieval model, uses the new fault cases as training samples to update the log semantic vector model, and optimizes the generation strategy of the large language model based on the feedback scores of the operation and maintenance personnel on the diagnostic report.

8. The intelligent fault location method for GPU servers based on the fusion of knowledge graph and large language model as described in claim 1, characterized in that: The method adopts a microservice architecture design, and each module can be expanded, combined or integrated with external systems as needed. It supports the connection with automated operation and maintenance platforms, alarm systems, monitoring platforms, equipment management systems and container scheduling systems to achieve a closed loop of linkage between multiple systems.

Images