A GPU node automatic fault detection and repair method and device

Abstract

This invention proposes an automatic fault detection and repair method and apparatus for GPU nodes. The method includes: real-time acquisition of hardware operation indicators, software status indicators, and resource load indicators of graphics processing unit nodes, and streaming the acquired indicator data to an analysis terminal; performing time-series analysis on the indicator data based on a machine learning model to predict potential fault risks, and comparing the indicator data against thresholds using a preset rule base to identify faults that have occurred, thereby generating fault diagnosis results; matching corresponding repair strategies from a multi-level self-healing strategy library according to the fault type and severity level, and generating execution instructions; responding to the execution instructions, performing corresponding isolation marking, lossless migration of computing tasks, system reset, or resource masking operations on the graphics processing unit nodes, and verifying the node status after the operation is completed to determine whether to restore the node to the cluster or trigger a manual maintenance notification.

Description

Technical Field

[0001] This invention belongs to the field of server technology, specifically relating to a method and apparatus for automatic fault detection and repair of GPU nodes. Background Technology

[0002] GPU clusters, as core infrastructure for artificial intelligence and high-performance computing, are widely used in deep learning training and scientific computing. With the expansion of cluster scale, a basic operation and maintenance system has been built through the collaborative operation of container orchestration platforms, real-time monitoring tools, and scheduling systems. Specifically, this system covers the entire process from metric collection to task scheduling, including key aspects such as hardware status monitoring, resource allocation management, and anomaly alerts.

[0003] However, existing fault management methods directly use static threshold judgment or periodic polling, without establishing a prediction model based on time-series characteristics and a multi-level self-healing mechanism. This may lead to a delay in identifying potential risks such as memory leaks, or in the event of a driver crash, it may only rely on manual isolation and lack automatic migration capabilities, thereby affecting the overall reliability of the cluster and the utilization of computing resources. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to propose an automatic fault detection and repair method for GPU nodes.

[0006] The second objective of this invention is to provide an automatic fault detection and repair device for GPU nodes.

[0007] The third objective of this invention is to provide a computer device.

[0008] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0009] To achieve the above objectives, a first aspect of the present invention provides an automatic fault detection and repair method for GPU nodes, comprising:

[0010] S1 collects hardware operation indicators, software status indicators, and resource load indicators of the graphics processing unit node in real time, and streams the collected indicator data to the analysis end. S2, perform time-series analysis on the indicator data based on the machine learning model to predict potential fault risks, and combine the indicator data with a preset rule base to perform threshold comparison to identify faults that have occurred, thereby generating fault diagnosis results that include fault type and severity level. S3. Based on the fault type and severity level in the fault diagnosis results, match the corresponding repair strategy from the multi-level self-healing strategy library and generate execution instructions including node isolation, task migration, node restart or node offline maintenance. S4, in response to the execution instruction, performs corresponding isolation marking, lossless migration of computing tasks, system reset or resource masking operations on the graphics processing unit node, and verifies the node status after the operation is completed to determine whether to restore the node to the cluster or trigger a manual maintenance notification.

[0011] In one embodiment of the present invention, S1 includes: Collect GPU temperature, power consumption, memory usage, error count, fan speed, and onboard voltage as hardware performance indicators; Collect driver loading status, container running status, and error codes as software status indicators; Collect GPU utilization, memory usage, and CPU load as resource load indicators; The collected indicator data is encapsulated into structured data and pushed to the analysis end in real time through a streaming transmission channel.

[0012] In one embodiment of the present invention, S2 includes: A time-series prediction model is constructed, the index data is input into the model for forward propagation calculation, and the predicted failure probability value for the future time window is output. Simultaneously, it iterates through the threshold rules in the preset rule base, and immediately marks a real-time fault when the video memory usage or temperature exceeds the threshold. By combining the predicted fault probability values ​​and real-time fault labeling results, predictive detection events and real-time detection events are distinguished, and a fault diagnosis result is generated based on the principle that real-time detection has a higher priority than predictive detection; wherein, The predictive detection is used to identify potential risks before a failure occurs, and the real-time detection is used to respond immediately to a failure that has already occurred.

[0013] In one embodiment of the present invention, S3 includes: When the fault type is determined to be a temporary software fault, the node restart strategy is matched and execution instructions for safe shutdown and power-on reset are generated; When the fault type is determined to be a temporary fault of resource bottleneck, the task migration and partial resource isolation strategies are matched to generate instructions to add a scheduling exclusion flag to the fault node and block the execution of abnormal devices. When the fault type is determined to be a permanent hardware fault and meets any of the following conditions: hardware error continues to exceed the threshold, restart repair fails, or hardware error is unrecoverable, the full node isolation and offline maintenance strategy is matched to generate an execution instruction that marks the node as unschedulable and evicts the remaining tasks.

[0014] In one embodiment of the present invention, S4 includes: When the execution instruction is a task migration, a checkpoint process is triggered to persist the computation state to shared storage; The scheduling algorithm, based on remaining resources and health status, selects healthy nodes as target nodes. The persistent state data is loaded to the target node and the computation task is resumed. If the recovery is successful, the fault mark of the source node is cleared; if it fails, a manual maintenance notification is triggered.

[0015] In one embodiment of the present invention, S4 further includes: After the system reset operation is completed, hardware diagnostics, software checks and resource verification steps are executed in sequence. If all verification steps pass, the node isolation mark will be automatically removed and the node will be re-added to the computing queue. If any verification step fails, it will be determined as an unhealable fault, the node will be taken offline, and an alarm message will be pushed to the operation and maintenance personnel. After a node is taken offline for maintenance, it responds to the recovery test command triggered by the operations and maintenance personnel by sequentially performing hardware self-test, environment verification, stress test and task trial run; If all the above tests pass, the node isolation mark will be automatically removed and the node will be rejoined to the cluster; otherwise, the node will remain offline and a further maintenance alert will be pushed.

[0016] To achieve the above objectives, a second aspect of the present invention provides an automatic fault detection and repair device for GPU nodes, comprising: The monitoring agent module is used to collect hardware operation indicators, software status indicators, and resource load indicators of the graphics processing unit nodes in real time, and to stream the collected indicator data to the analysis end. The analysis engine module is used to perform time-series analysis on the indicator data based on a machine learning model to predict potential failure risks, and to perform threshold comparison on the indicator data in conjunction with a preset rule base to identify failures that have occurred, thereby generating a failure diagnosis result that includes the failure type and severity level. The decision controller module is used to match the corresponding repair strategy from the multi-level self-healing strategy library according to the fault type and severity level in the fault diagnosis results, and generate execution instructions including node isolation, task migration, node restart or node offline maintenance. The scheduler module is used to respond to the execution command by performing corresponding isolation marking, lossless migration of computing tasks, system reset or resource masking operations on the graphics processing unit node, and verifying the node status after the operation is completed to determine whether to restore the node to the cluster or trigger a manual maintenance notification.

[0017] An automatic fault detection and repair method and apparatus for GPU nodes according to an embodiment of the present invention has the following beneficial effects: This solution utilizes automated mechanisms such as real-time monitoring, automatic repair, machine learning prediction, and task migration and retries to reduce fault handling time from hours to minutes or even seconds. For example, node restart and verification can be completed within 5 minutes, far superior to the more than 30 minutes required for manual operation, significantly reducing the mean time to repair (MTTR) and improving cluster availability. Simultaneously, predictive detection and rapid recovery reduce node downtime, increasing GPU cluster resource utilization from 60% to over 85%. Task migration and retry mechanisms ensure seamless and continuous operation of long-term computing tasks such as deep learning, reducing task failure rates and improving computational efficiency. Full-process automation significantly reduces manual intervention and operational pressure, eliminating the need for 24 / 7 fault handling and effectively reducing human error and operational costs. The solution can be deeply integrated with container orchestration platforms such as Kubernetes, adapting to physical machines, virtual machines, and cloud environments, effectively addressing the challenges of dynamic scheduling. Furthermore, machine learning models enable potential fault prediction and preventative maintenance, avoiding emergency handling after a fault occurs and further enhancing overall cluster stability.

[0018] To achieve the above objectives, a third aspect of this application provides a computer device, including a processor and a memory; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, for implementing an automatic fault detection and repair method for GPU nodes as described in the first aspect embodiment.

[0019] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements an automatic fault detection and repair method for GPU nodes as described in the first aspect embodiment.

[0020] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of an automatic fault detection and repair method for GPU nodes according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall architecture of an automatic fault detection and repair system for GPU nodes according to an embodiment of the present invention; Figure 3This is an architecture diagram of an automatic fault detection and repair method for GPU nodes according to an embodiment of the present invention; Figure 4 This is a structural diagram of an automatic fault detection and repair device for GPU nodes according to an embodiment of the present invention; Figure 5 It is a computer device according to an embodiment of the present invention. Detailed Implementation

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] The following description, with reference to the accompanying drawings, describes a method and apparatus for compressing and storing historical data of a hydropower monitoring system according to an embodiment of the present invention.

[0025] Example 1 Figure 1 This is a flowchart of a method for compressing and storing historical data in a hydropower monitoring system according to an embodiment of the present invention, such as... Figure 1 As shown, it includes: S1 collects hardware operation indicators, software status indicators, and resource load indicators of the graphics processing unit nodes in real time, and streams the collected indicator data to the analysis end.

[0026] Specifically, the hardware operation metrics, software status metrics, and resource load metrics of the graphics processing unit nodes are collected in real time, and the collected metric data is streamed to the analysis end. This step aims to build a highly timely cluster status awareness foundation. Through data acquisition units deployed locally on the nodes, multi-dimensional features reflecting node health are periodically acquired. The hardware operation metrics cover parameters reflecting physical layer stability, the software status metrics cover parameters reflecting the integrity of the driver and runtime environment, and the resource load metrics cover parameters reflecting the allocation of computing power. The acquisition process emphasizes low latency and continuity. By encapsulating discrete sampling points into structured data streams, and using bidirectional streaming communication protocols or message queue mechanisms, the data is proactively pushed to the remote analysis end, thereby avoiding the response lag caused by traditional polling methods and ensuring that the analysis end can perform fault assessment based on the latest status sequence.

[0027] As a specific implementation method, specific indicators such as temperature, power consumption, memory usage, PCIe error count, driver status, and CUDA runtime status can be collected using NVML, nvidia-smi, or IPMI interfaces, and reported in real time using gRPC bidirectional stream or Kafka message queue in Protobuf serialization format. The collection period can be set to the second level.

[0028] This step significantly improves the timeliness and accuracy of fault detection by establishing a multi-dimensional real-time data perception channel, providing a high-quality data foundation for subsequent predictive maintenance. The streaming mechanism effectively eliminates the time lag between data acquisition and analysis, enabling the system to capture abnormal features in the early stages of a fault or even before it occurs, thereby greatly shortening the mean time to repair and ensuring the continuity of computational tasks.

[0029] Furthermore, in this embodiment, the specific implementation process of step S1 is as follows: The monitoring agent first uses NVML library function calls, nvidia-smi command execution, and IPMI protocol communication as input sources to read GPU core temperature sensor values, instantaneous power consumption sensor values, video memory usage, and PCIe link error count register values. At the same time, it reads fan speed register data and onboard voltage ADC sampling values ​​recorded by the board management controller through the IPMI protocol to form a raw data set of hardware operation indicators.

[0030] Next, the system timestamps the raw sensor data read above with the PCIe link error count register value to eliminate time deviations in multi-source data acquisition and generate a hardware operation index data stream with a unified time stamp. This data stream serves as the basic input data for subsequent time series analysis.

[0031] Meanwhile, the monitoring agent uses CUDA runtime interface calls and Kubernetes API queries as input sources to obtain driver loading status, container running status, and XID error codes as software status indicators, and uses system proc file system reads as input sources to obtain GPU utilization, host memory usage, and CPU load as resource load indicators.

[0032] Subsequently, the processing action encapsulates the aforementioned multi-source heterogeneous hardware operation indicator data streams, software status indicators, and resource load indicators into structured data in Protobuf serialization format, ensuring efficient and compatible data transmission.

[0033] Finally, using gRPC bidirectional streaming channels or Kafka message queues as transmission media, the encapsulated structured data is pushed to the analysis end in real time at a frequency of once per second, completing a complete closed loop from data collection, formatting and encapsulation to streaming transmission, providing the analysis engine with continuous, synchronous and standardized real-time monitoring data input.

[0034] S2, perform time-series analysis on the indicator data based on a machine learning model to predict potential fault risks, and combine the indicator data with a preset rule base to perform threshold comparison to identify faults that have occurred, thereby generating fault diagnosis results that include fault type and severity level.

[0035] Specifically, this step aims to achieve a fault identification mechanism that moves from passive response to proactive prevention through in-depth processing of multi-dimensional indicator data. Its core lies in constructing a dual analytical architecture that integrates data-driven prediction and rule-driven detection. Machine learning models are used to mine the evolutionary patterns of indicator data over time to capture potential fault risks that have not yet manifested but have a clear development trend. Simultaneously, a pre-defined rule base is used to logically compare the indicator values ​​at the current moment to quickly identify substantive faults that have already occurred. This collaborative working mode of time-series analysis and threshold comparison can comprehensively cover different stages of the fault lifecycle, ensuring the completeness and timeliness of the diagnostic results. The analysis process ultimately outputs standardized fault diagnosis results, which at least include a classification definition of the essential attributes of the fault and a quantitative rating of its impact, providing a precise basis for subsequent strategy matching.

[0036] As a specific implementation method, a long short-term memory network can be used as a machine learning model to extract features from time-series data such as temperature, power consumption and memory usage to predict memory leaks or heat dissipation anomalies. Combined with a rule base containing specific threshold conditions, driver crashes or hardware errors can be identified, thereby distinguishing the priority of predictive detection and real-time detection and generating corresponding fault types and severity levels.

[0037] By introducing machine learning models for time-series analysis, this method overcomes the limitations of traditional single-threshold judgments, enabling early identification of slowly developing latent faults and significantly improving the accuracy and foresight of fault prediction. Simultaneously, the real-time comparison mechanism based on a pre-defined rule base ensures rapid response to sudden, already occurring faults. These two complementary approaches form a round-the-clock fault awareness system, effectively reducing the missed detection rate and false alarm rate. This provides reliable data support for subsequent implementation of differentiated multi-level self-healing strategies, thereby significantly shortening the average fault repair time and ensuring the continuity of cluster services.

[0038] Furthermore, in this embodiment, the specific implementation process of step S2 is as follows: First, in the model building and data input phase, the input source is 16 time-series feature vectors collected by the monitoring agent, including GPU temperature, power consumption, memory usage, and PCIe error count. The processing involves constructing the above indicator data into an input tensor using a sliding window mechanism with a time step size of 10, and then feeding it into a Long Short-Term Memory (LSTM) network model. This model is specifically configured with 256 hidden layer units, using a batch size of 32 for batch inference computation, and updating the network weights using backpropagation with a learning rate of 0.001. The output is a predicted failure probability value for the future time window, representing the trends in memory leakage, temperature drift, and the risk of slow frequency reduction. This value serves as the core basis for predictive detection.

[0039] Secondly, in the real-time rule determination phase, the input source is the real-time indicator data at the current moment. The processing action involves traversing the threshold rules in the preset rule base to specifically determine whether the video memory usage exceeds 95% or whether the GPU temperature exceeds [a certain threshold]. Once any metric is detected to exceed the threshold, it is immediately marked as a real-time fault. The output is a real-time fault marker signal with a timestamp.

[0040] Finally, in the diagnostic result generation stage, the input sources are the aforementioned fault probability prediction values ​​and real-time fault labeling results. The processing action involves executing priority arbitration logic, following the principle that real-time detection has higher priority than predictive detection. That is, when real-time fault labels exist, the predicted value is directly ignored and the fault is determined to be of a high severity level; only when there are no real-time fault labels is the potential risk level determined based on the fault probability prediction value. The output result is the final fault diagnosis result containing a clear fault type and severity level, which will directly trigger subsequent multi-level self-healing strategy matching.

[0041] By employing a Long Short-Term Memory (LSTM) network model with 16 input dimensions, 10 time steps, and 256 hidden layers, this embodiment can accurately capture the temporal dependencies of GPU performance metrics and effectively identify progressive fault risks such as memory leaks. Simultaneously, by combining real-time threshold comparison and priority arbitration mechanisms, it ensures both a second-level response to sudden hardware anomalies and proactive warnings of potential hazards, significantly improving the accuracy and timeliness of fault detection and avoiding the problems of missed or false alarms caused by a single detection mode.

[0042] S3. Based on the fault type and severity level in the fault diagnosis results, match the corresponding repair strategy from the multi-level self-healing strategy library and generate execution instructions including node isolation, task migration, node restart, or node offline maintenance.

[0043] This step aims to address the technical problem of existing fault repair strategies being singular and lacking adaptability. Its core lies in constructing a decision-making mechanism that dynamically matches multi-level self-healing strategies based on fault characteristics. Specifically, the process receives fault diagnosis results containing fault type and severity level, uses them as decision input variables, and executes mapping logic within a pre-defined multi-level self-healing strategy library to generate targeted execution instructions. This multi-level self-healing strategy library stores the correlation between different fault scenarios and repair actions, and can adaptively select one or more combinations of operations from node isolation, task migration, node restart, or node offline maintenance based on attributes such as the temporary nature of the fault, resource bottleneck nature, or permanent hardware damage. This matching process does not rely on a fixed linear flow but dynamically adjusts the repair intensity according to the fault severity level to achieve a tiered response from lightweight reset to heavy-duty offline.

[0044] As one implementation method, when the fault diagnosis result indicates a temporary software anomaly, a node restart strategy can be matched; when it indicates a resource bottleneck, a task migration strategy can be matched; and when it is determined to be an unrecoverable hardware failure, a node isolation and offline maintenance strategy is matched, and corresponding taint marks or eviction instructions are generated through the container orchestration interface.

[0045] By employing a dynamic policy matching mechanism based on fault type and severity level, this step avoids the resource waste or insufficient repair caused by the "one-size-fits-all" approach in traditional technologies, significantly improving the cluster's adaptive handling capability for diverse faults. Simultaneously, the flexible invocation of multi-level self-healing strategies ensures rapid fault closure with minimal system overhead while maintaining business continuity, effectively reducing average repair time and improving overall resource utilization.

[0046] Furthermore, in this embodiment, the specific implementation process of step S3 is as follows: When the fault diagnosis results output by the analysis engine indicate that the fault type is a temporary software fault or driver crash, the decision controller receives the fault type and severity level as input, immediately matches the node restart strategy, and the processing action is to construct a control package containing a safe shutdown sequence and a power-on reset command, and output the execution command sent through the IPMI interface or BMC hardware management interface to trigger the system reset of the faulty node.

[0047] When the fault diagnosis result indicates that the fault type is a temporary fault of resource bottleneck, the decision controller receives the abnormal GPU device identifier and resource load data as input, matches the task migration and partial resource isolation policies, and the processing action is to call the Kubernetes device plugin interface to add the device.kubernetes.io / gpu-unhealthy taint to the identified abnormal GPU device to achieve device-level scheduling exclusion, while maintaining the normal allocation status of other healthy GPUs and CPU resources in the same node, and generating configuration instructions containing computing power frequency lock values ​​and memory usage quota limit parameters. The device-level isolation instruction that only applies computing power lock and memory quota limit to the abnormal GPU is output to the scheduler, thereby completing fine-grained resource masking without affecting the overall availability of the node.

[0048] When the fault diagnosis result indicates that the fault type is a permanent hardware fault, and the judgment conditions are met, such as the GPU hardware error count continuously exceeding the preset threshold, automatic restart repair failure, or unrecoverable memory and PCIe hardware errors, the decision controller receives the fault node ID and historical repair records as input, matches the full node isolation and offline maintenance strategy, and the processing action is to generate a standard instruction to mark the node as unschedulable and calculate the remaining task eviction list, output an execution package containing node taint marking operation and task migration instructions, forcibly stop all resource allocation entry points of the node, and notify the operation and maintenance personnel to intervene.

[0049] In each of the above sub-steps, the instruction parameters output by the previous strategy matching stage are directly used as the input control quantities for the next scheduling execution stage, ensuring the accuracy of instruction generation and the continuity of execution.

[0050] This implementation method, by subdividing fault types and matching differentiated repair instructions, achieves rapid restart and recovery from temporary software faults, fine-grained device-level isolation of resource bottlenecks, and complete offline handling of permanent hardware faults. It effectively avoids resource waste or fault propagation caused by a single repair strategy, and significantly improves the self-healing accuracy and overall resource utilization of the GPU cluster.

[0051] S4, in response to the execution instruction, performs corresponding isolation marking, lossless migration of computing tasks, system reset or resource masking operations on the graphics processing unit node, and verifies the node status after the operation is completed to determine whether to restore the node to the cluster or trigger a manual maintenance notification.

[0052] In response to the generated execution instructions, this step performs repair operations by implementing differentiated physical or logical interventions on the graphics processing unit nodes. Its core lies in dynamically calling the underlying management interface according to the instruction type to complete the change of node status and the reconstruction of resources.

[0053] Specifically, the isolation marking operation injects specific exclusion attributes into the cluster scheduling system to block the allocation path of new computing tasks to abnormal nodes; the lossless migration operation of computing tasks involves establishing a state synchronization channel between the source node and the target node to completely transfer the running computing context to the healthy node to ensure business continuity; the system reset operation triggers the node restart process through the hardware management interface or operating system kernel instructions to clear transient software anomalies; and the resource masking operation targets certain hardware failures by disabling abnormal computing units at the device level while preserving the availability of the remaining resources of the node.

[0054] After any of the above operations are completed, the system automatically enters the status verification phase. By re-collecting hardware diagnostic data, software service status, and resource load indicators, the system assesses whether the node meets the health standards of the regression cluster. If the verification is successful, the isolation is automatically lifted and the node is re-added to the scheduling queue. If the verification fails, it is determined to be a self-healing fault and a manual maintenance notification is triggered.

[0055] As one implementation method, isolation can be achieved by adding taint marks to faulty nodes through the Kubernetes API, and task migration can be completed using the joint mechanism of CUDA checkpoints and container snapshots. Alternatively, a hardware reset can be performed through the IPMI interface, and memory detection and driver verification can be performed sequentially after restarting to determine the node's destination.

[0056] This step, through a closed-loop execution and verification mechanism, automates and intelligentizes the fault repair process, significantly shortening the average repair time of nodes. At the same time, by leveraging lossless migration and fine-grained resource shielding technologies, it maximizes the continuity of computing tasks and the utilization rate of overall cluster resources, effectively reducing the risk of business interruption caused by single point of failure.

[0057] Furthermore, in this embodiment, the specific implementation process of step S4 is as follows: When the execution instruction generated by the decision controller is a task migration, the system first receives the task migration trigger signal as input, then pauses the currently running deep learning training process, calls the CUDA runtime interface to capture the model weight data and optimizer state in the GPU memory, and simultaneously calls the container engine interface to perform a snapshot operation on the running container file system to capture the application binary file and dependency library state. Then, the model weight data, optimizer state, training step counter, and container snapshot data are packaged and encrypted, and written in parallel to the distributed shared storage system via a high-speed network to generate a checkpoint file containing complete context information. The path index of this checkpoint file is used as the input parameter for subsequent scheduling.

[0058] Next, the scheduler takes the path index and the real-time resource status of the cluster as input, executes a weighted scheduling algorithm based on remaining resources and health, traverses the cluster nodes, and selects healthy nodes with GPU utilization below 70% and remaining video memory greater than 20% as target nodes, outputting the target node identifier and resource allocation scheme. Subsequently, the target node loads the persistent status data according to the allocation scheme and resumes the execution of the computing task. If the task recovery is successful, a success signal is output and the fault mark of the source node is cleared; if the task recovery fails, the system starts a retry mechanism, taking the failure status as input, and retrying loading and recovery after a 30-second interval between each retry, accumulating the number of retries. When the number of retries reaches 3 and still fails, the final failure result is output and a manual maintenance notification is triggered. The log and notification system pushes alarm information containing the faulty node ID, fault type, and historical repair records to the operation and maintenance personnel, thereby completing the closed-loop processing from checkpoint generation to migration verification.

[0059] The above implementation achieves lossless persistence of deep learning task states through the combined application of the CUDA Checkpoint mechanism and container snapshots. Combined with a fine-grained scheduling strategy based on resource utilization and GPU memory availability, it ensures seamless migration and rapid recovery of computing tasks in the event of node failure, effectively avoiding interruption and progress loss of long-cycle training tasks, and significantly improving the business continuity and resource utilization efficiency of GPU clusters.

[0060] Example 2 This invention provides an automatic fault detection and repair method for GPU nodes, aiming to solve the problems of lagging fault detection, limited repair strategies, and lack of intelligent prediction in existing technologies. This method automatically detects potential faults (such as hardware failures, software anomalies, or resource bottlenecks) by monitoring the hardware status, software operation, and resource usage of GPU nodes in real time, and initiates a multi-level self-healing mechanism to execute the repair process. The ultimate goal is to reduce manual intervention and improve cluster reliability and resource utilization.

[0061] The core architecture of this invention includes the following modules: Monitoring Agent: Deployed on each GPU node, the monitoring agent is responsible for collecting hardware metrics (such as temperature, power consumption, and memory usage), software status (such as drivers and CUDA runtime status), and resource usage data (such as memory and compute load). The monitoring agent transmits data to the analytics engine in real time via streaming processing.

[0062] Analytics Engine: Analyzes monitoring data using machine learning models (such as Long Short-Term Memory (LSTM) networks or anomaly detection algorithms) to predict potential faults. The analytics engine also integrates a rule base to support threshold-based detection and pattern recognition.

[0063] Decision controller: Triggers corresponding remediation strategies (such as isolation, migration, restart, or offline maintenance) based on the fault type and severity.

[0064] Scheduler: Integrates with container orchestration platforms (such as Kubernetes) and is responsible for performing task migration, node isolation, and resource rescheduling.

[0065] Log and notification system: Records fault events and repair operations, and notifies operations and maintenance personnel to participate in the handling of faults that cannot be automatically repaired.

[0066] The calling relationships between the modules are as follows, and all modules use JSON format for data transfer.

[0067] Monitoring Agent → Analysis Engine: The monitoring agent actively collects metric data and sends it to the analysis engine via streaming push; the analysis engine passively receives the data and does not pull it back.

[0068] Analysis Engine → Decision Controller: After the analysis engine completes fault detection / prediction, it pushes the fault type, severity level, fault node ID, and abnormal indicators to the decision controller to trigger policy decisions.

[0069] Decision Controller → Scheduler: The decision controller issues isolation / migration / restart / offline commands based on the fault level, and the scheduler executes the cluster-level operations and returns the execution results.

[0070] Scheduler / Decision Controller → Log and Notification System: All fault events, repair actions, and execution status are written to the log system in real time; faults that cannot be automatically repaired are alerted by the notification system.

[0071] Deployment locations of each module: Monitoring agent deployment location: Local on each GPU compute node (physical machine / virtual machine / container); Deployment method: DaemonSet (K8s) / system service; Analysis engine deployment location: Cluster management node (Master node), supporting clustered deployment; Decision controller deployment location: Cluster management node, on the same machine as the analytics engine or in a separate pod; Scheduler deployment location: Kubernetes control plane, integrated as an extended scheduler; Log and notification system deployment location: cluster management node / standalone log server.

[0072] Table 1: Main Modules and Functions of the Invention

[0073] The workflow of this invention is as follows: Data collection: The monitoring agent collects various metrics of the GPU nodes in real time.

[0074] Fault detection: The analysis engine analyzes the data to identify potential faults.

[0075] Fault Classification: Faults are classified according to their type (hardware, software, or resource bottleneck) and severity.

[0076] Repair execution: The decision controller triggers the corresponding repair operations, including isolation, migration, restart, or offline maintenance.

[0077] Verification and Recovery: After repair, verify the node status and rejoin the cluster once it is confirmed to be normal.

[0078] Appendix Figure 2 This is a schematic diagram of the overall architecture of the GPU node automatic fault detection and repair system. It clearly shows that the system consists of five core modules: monitoring agent, analysis engine, decision controller, scheduler, and log and notification system. It fully presents the automated data flow and calling relationship of the entire process from GPU node indicator collection, fault analysis and prediction, repair strategy decision-making, cluster scheduling execution to log alarms and status verification, realizing real-time detection, intelligent prediction, multi-level self-healing and closed-loop recovery of GPU cluster faults.

[0079] Among them, (1) the monitoring agent module is responsible for collecting hardware indicators, software status, and resource usage data of GPU nodes in real time. (2) The monitoring agent module transmits the collected data to the analysis engine module. (3) The analysis engine module receives and parses the monitoring data, performs time-series prediction through machine learning models, performs threshold detection in combination with rule bases, completes fault identification and level classification, and outputs standardized fault information to the decision controller. (4) The decision controller module dynamically matches multi-level self-healing strategies according to the fault type and severity, generates execution instructions such as isolation, task migration, node restart, and offline maintenance, sends them to the scheduler, and receives the execution results for status verification. At the same time, it sends relevant events to the log and notification system for recording. (5) The scheduler module executes the scheduling instructions issued by the decision controller, completes fault node taint marking, task migration and status saving, and resource reallocation, and feeds back the results to the log and notification system after execution. (6) The log and notification system module records the collected data, fault events, repair operations, and execution results in real time; pushes alarms to maintenance personnel for faults that cannot be automatically repaired, and provides full-process traceable logs.

[0080] like Figure 3 As shown, the implementation method of this embodiment includes the following steps: Fault detection: The monitoring agent is deployed on each GPU node, and the monitoring metrics collected and their corresponding collection methods are as follows: Hardware metrics include GPU temperature, power consumption, memory usage, PCIe error count, fan speed, and onboard voltage, obtained through NVML, nvidia-smi, and IPMI / BMC; Software metrics include driver status, CUDA runtime status, container runtime status, and XID errors, obtained through NVML, nvidia-smi, CUDA interface, and Kubernetes API; Resource metrics include GPU utilization, host memory usage, and CPU load, obtained through NVML and the system proc file system, with a collection period of 1 second.

[0081] Streaming implementation: The monitoring agent encapsulates the collected metric data into structured data and uses gRPC bidirectional streaming or Kafka message queues to achieve low-latency real-time reporting. The data is sent to the analysis engine in Protobuf serialization format to ensure efficient and reliable transmission.

[0082] The specific parameters of the LSTM model are as follows: input dimension is 16-dimensional monitoring indicators, time step is 10, number of hidden layer units is 256, batch size is 32, learning rate is 0.001, and it is used for feature extraction of time series data and fault trend prediction.

[0083] Distinguishing between predictive and real-time detection: Predictive detection, based on historical time-series data, uses an LSTM model to identify potential fault risks such as memory leaks and continuous temperature increases, acting as a proactive warning. Real-time detection, based on current metrics and a rule base, immediately identifies faults that have already occurred, such as driver crashes and hardware errors, acting as an in-process response. Priority of detection results: Real-time detection has higher priority than predictive detection. Real-time faults directly trigger the repair process, while predictive faults only trigger warnings and preventative measures.

[0084] Data Collection: The monitoring agent periodically (e.g., every second) collects metric data from the GPU nodes, including: Hardware metrics: temperature (obtained via NVML), power consumption, memory usage, and PCIe error count.

[0085] Software metrics: Driver status (queried via nvidia-smi), CUDA runtime status, container runtime status.

[0086] Resource metrics: GPU utilization, memory usage, and compute load. Data is streamed to the analytics engine to avoid latency.

[0087] Fault Prediction and Detection: The analytics engine uses machine learning models (such as LSTM) to analyze time-series data and predict potential faults (e.g., a sustained increase in temperature may indicate a heat dissipation problem). Simultaneously, a rule base triggers immediate alerts based on thresholds (e.g., memory usage exceeding 95% or temperature exceeding 85°C). Fault detection is divided into two categories: predictive detection: identifying risks before a fault occurs, such as predicting memory leaks using historical data; and real-time detection: responding immediately to faults that have already occurred (e.g., driver crashes).

[0088] Fault isolation: Fault isolation is achieved through taint labeling, hierarchical isolation strategies, and fault type determination, as detailed below: The specific steps for setting taints are as follows: The decision controller adds the node.kubernetes.io / gpu-fault taint to the faulty node through the Kubernetes API Server. The effect is NoSchedule, which prevents new Pods from being scheduled to that node. At the same time, the device.kubernetes.io / gpu-unhealthy taint is added to the faulty GPU device to achieve device-level scheduling exclusion.

[0089] Partial resource isolation implementation: For temporary failures caused by resource bottlenecks, fine-grained isolation at the device level is adopted. Only the abnormal GPU is locked in terms of computing power and memory quota is limited. The abnormal GPU is also blocked through device plugins. Other GPU and CPU resources on the node can still be allocated and used normally.

[0090] Permanent Fault Judgment Criteria: A permanent fault is determined if any of the following conditions are met: GPU hardware error count continuously exceeds the threshold, automatic restart repair fails, memory / PCIe hardware errors are unrecoverable, or key indicators such as temperature / voltage continuously exceed safe ranges.

[0091] Task migration and retry: Task migration and retry consist of three parts: checkpoint mechanism, rescheduling, and automatic retry. The checkpoint mechanism is implemented through a combination of CUDA Checkpoint and container snapshots. Before task migration, the model weights, training steps, and intermediate computation states are persisted to shared storage and automatically executed when task migration is triggered. The rescheduling adopts a weighted scheduling algorithm based on remaining resources and health, giving priority to healthy nodes with GPU utilization <70%, remaining video memory >20%, and no fault markers, and allocating target nodes according to the principles of resource matching and load balancing; The retry mechanism is set to a maximum of 3 retries, with a 30-second interval between each retry. Retry is only performed for software exceptions and temporary resource shortages. Hardware failures are not retried. An alarm is triggered if a retry fails.

[0092] Node restart and detection: Automatic restart is achieved through IPMI, BMC hardware management interface, or operating system remote call interface. After the decision controller issues a restart command, the node performs a safe shutdown and power-on reset. After restarting, it enters the verification step, which performs hardware diagnostics, software checks, and resource verification in sequence. Hardware diagnostics include GPU path, video memory, and PCIe link detection. Software checks include driver loading, CUDA runtime, and container service status verification. Resource verification includes whether indicators such as GPU utilization, temperature, and power consumption have returned to the normal range. If the verification fails after restarting, the system determines that it is a fault that cannot be self-healed, immediately upgrades the node to a completely isolated state, stops all repair attempts, automatically triggers the node offline process and synchronizes it to the log and notification system, and pushes operation and maintenance alarms.

[0093] For restartable and recoverable failures (such as software crashes or temporary resource exhaustion), the decision controller triggers an automatic restart (via IPMI or BMC interface). After restarting, the monitoring agent verifies the node status: Hardware diagnostics: Run basic hardware tests.

[0094] Software check: Verify the status of drivers and services.

[0095] Resource verification: Confirms that resources are being used normally. After successful verification, the node automatically returns to the computation queue.

[0096] Offline repair and restoration: The automatic shutdown operation is initiated by the decision controller via the Kubernetes API, marking the node as unschedulable, simultaneously evicting remaining tasks on the node, blocking all resource allocation entry points, and updating the cluster node status database to "under maintenance." The system then notifies operations personnel via email, SMS, and platform in-system messaging, automatically pushing the faulty node ID, fault type, historical repair records, and shutdown time to the operations team. Recovery testing begins after manual repair, sequentially executing GPU hardware self-test, driver and CUDA environment verification, stress testing, and business task trial run. Once all tests pass, the isolation is lifted, and the node reconnects to the cluster.

[0097] For faults that cannot be automatically repaired (such as hardware failure), the decision controller will automatically take the node offline and notify the operations and maintenance personnel for repair. After repair, the operations and maintenance personnel will trigger a recovery test through the management interface to verify the node's stability. Once confirmed to be normal, the node will be rejoined to the cluster.

[0098] Example 3 To achieve the above embodiments, such as Figure 4 As shown, this embodiment also provides a historical data compression and storage device 10 for a hydropower monitoring system. The device 10 includes a monitoring agent module 100, an analysis engine module 200, a decision controller module 300, and a scheduler module 400.

[0099] The monitoring agent module 100 is used to collect hardware operation indicators, software status indicators and resource load indicators of the graphics processing unit node in real time, and to stream the collected indicator data to the analysis end. The analysis engine module 200 is used to perform time-series analysis on the indicator data based on a machine learning model to predict potential fault risks, and to perform threshold comparison on the indicator data in combination with a preset rule base to identify faults that have occurred, thereby generating fault diagnosis results that include fault type and severity level. The decision controller module 300 is used to match the corresponding repair strategy from the multi-level self-healing strategy library according to the fault type and severity level in the fault diagnosis result, and generate execution instructions including node isolation, task migration, node restart or node offline maintenance. The scheduler module 400 is used to respond to the execution command by performing corresponding isolation marking, lossless migration of computing tasks, system reset or resource masking operations on the graphics processing unit node, and verifying the node status after the operation is completed to determine whether to restore the node to the cluster or trigger a manual maintenance notification.

[0100] Furthermore, the aforementioned monitoring agent module 100 is also used for: Collect GPU temperature, power consumption, memory usage, error count, fan speed, and onboard voltage as hardware performance indicators; Collect driver loading status, container running status, and error codes as software status indicators; Collect GPU utilization, memory usage, and CPU load as resource load indicators; The collected indicator data is encapsulated into structured data and pushed to the analysis end in real time through a streaming transmission channel.

[0101] Furthermore, the aforementioned analysis engine module 200 is also used for: A time-series prediction model is constructed, the index data is input into the model for forward propagation calculation, and the predicted failure probability value for the future time window is output. Simultaneously, it iterates through the threshold rules in the preset rule base, and immediately marks a real-time fault when the video memory usage or temperature exceeds the threshold. By combining the predicted fault probability values ​​and real-time fault labeling results, predictive detection events and real-time detection events are distinguished, and a fault diagnosis result is generated based on the principle that real-time detection has a higher priority than predictive detection; wherein, The predictive detection is used to identify potential risks before a failure occurs, and the real-time detection is used to respond immediately to a failure that has already occurred.

[0102] Furthermore, the decision controller module 300 described above is also used for: When the fault type is determined to be a temporary software fault, the node restart strategy is matched and execution instructions for safe shutdown and power-on reset are generated; When the fault type is determined to be a temporary fault of resource bottleneck, the task migration and partial resource isolation strategies are matched to generate instructions to add a scheduling exclusion flag to the fault node and block the execution of abnormal devices. When the fault type is determined to be a permanent hardware fault and meets any of the following conditions: hardware error continues to exceed the threshold, restart repair fails, or hardware error is unrecoverable, the full node isolation and offline maintenance strategy is matched to generate an execution instruction that marks the node as unschedulable and evicts the remaining tasks.

[0103] Furthermore, the scheduler module 400 described above is also used for: When the execution instruction is a task migration, a checkpoint process is triggered to persist the computation state to shared storage; The scheduling algorithm, based on remaining resources and health status, selects healthy nodes as target nodes. The persistent state data is loaded to the target node and the computation task is resumed. If the recovery is successful, the fault mark of the source node is cleared; if it fails, a manual maintenance notification is triggered.

[0104] Furthermore, the scheduler module 400 described above is also used for: After the system reset operation is completed, hardware diagnostics, software checks and resource verification steps are executed in sequence. If all verification steps pass, the node isolation mark will be automatically removed and the node will be re-added to the computing queue. If any verification step fails, it will be determined as an unhealable fault, the node will be taken offline, and an alarm message will be pushed to the operation and maintenance personnel. After a node is taken offline for maintenance, it responds to the recovery test command triggered by the operations and maintenance personnel by sequentially performing hardware self-test, environment verification, stress test and task trial run; If all the above tests pass, the node isolation mark will be automatically removed and the node will be rejoined to the cluster; otherwise, the node will remain offline and a further maintenance alert will be pushed.

[0105] To implement the methods of the above embodiments, the present invention also provides a computer device, such as... Figure 5As shown, the computer device 600 includes a memory 601 and a processor 602; wherein, the processor 602 reads executable program code stored in the memory 601 to run a program corresponding to the executable program code, so as to implement the various steps of the method described above.

[0106] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0107] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0108] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for automatic fault detection and repair of GPU nodes, characterized in that, include: S1 collects hardware operation indicators, software status indicators, and resource load indicators of the graphics processing unit node in real time, and streams the collected indicator data to the analysis end. S2, perform time-series analysis on the indicator data based on the machine learning model to predict potential fault risks, and combine the indicator data with a preset rule base to perform threshold comparison to identify faults that have occurred, thereby generating fault diagnosis results that include fault type and severity level. S3. Based on the fault type and severity level in the fault diagnosis results, match the corresponding repair strategy from the multi-level self-healing strategy library and generate execution instructions including node isolation, task migration, node restart or node offline maintenance. S4, in response to the execution instruction, performs corresponding isolation marking, lossless migration of computing tasks, system reset or resource masking operations on the graphics processing unit node, and verifies the node status after the operation is completed to determine whether to restore the node to the cluster or trigger a manual maintenance notification.

2. The method as described in claim 1, characterized in that, S1 includes: Collect GPU temperature, power consumption, memory usage, error count, fan speed, and onboard voltage as hardware performance indicators; Collect driver loading status, container running status, and error codes as software status indicators; Collect GPU utilization, memory usage, and CPU load as resource load indicators; The collected indicator data is encapsulated into structured data and pushed to the analysis end in real time through a streaming transmission channel.

3. The method as described in claim 1, characterized in that, The S2 includes: A time-series prediction model is constructed, the index data is input into the model for forward propagation calculation, and the predicted failure probability value for the future time window is output. Simultaneously, it iterates through the threshold rules in the preset rule base, and immediately marks a real-time fault when the video memory usage or temperature exceeds the threshold. By combining the predicted fault probability values ​​and real-time fault labeling results, predictive detection events and real-time detection events are distinguished, and a fault diagnosis result is generated based on the principle that real-time detection has a higher priority than predictive detection; wherein, The predictive detection is used to identify potential risks before a failure occurs, and the real-time detection is used to respond immediately to a failure that has already occurred.

4. The method as described in claim 1, characterized in that, The S3 includes: When the fault type is determined to be a temporary software fault, the node restart strategy is matched and execution instructions for safe shutdown and power-on reset are generated; When the fault type is determined to be a temporary fault of resource bottleneck, the task migration and partial resource isolation strategies are matched to generate instructions to add a scheduling exclusion flag to the fault node and block the execution of abnormal devices. When the fault type is determined to be a permanent hardware fault and meets any of the following conditions: hardware error continues to exceed the threshold, restart repair fails, or hardware error is unrecoverable, the full node isolation and offline maintenance strategy is matched to generate an execution instruction that marks the node as unschedulable and evicts the remaining tasks.

5. The method as described in claim 1, characterized in that, The S4 includes: When the execution instruction is a task migration, a checkpoint process is triggered to persist the computation state to shared storage; The scheduling algorithm, based on remaining resources and health status, selects healthy nodes as target nodes. The persistent state data is loaded to the target node and the computation task is resumed. If the recovery is successful, the fault mark of the source node is cleared; if it fails, a manual maintenance notification is triggered.

6. The method as described in claim 1, characterized in that, The S4 further includes: After the system reset operation is completed, hardware diagnostics, software checks and resource verification steps are executed in sequence. If all verification steps pass, the node isolation mark will be automatically removed and the node will be re-added to the computing queue. If any verification step fails, it will be determined as an unhealable fault, the node will be taken offline, and an alarm message will be pushed to the operation and maintenance personnel. After a node is taken offline for maintenance, it responds to the recovery test command triggered by the operations and maintenance personnel by sequentially performing hardware self-test, environment verification, stress test and task trial run; If all the above tests pass, the node isolation mark will be automatically removed and the node will be rejoined to the cluster; otherwise, the node will remain offline and a further maintenance alert will be pushed.

7. An automatic fault detection and repair device for GPU nodes, characterized in that, include: The monitoring agent module is used to collect hardware operation indicators, software status indicators, and resource load indicators of the graphics processing unit nodes in real time, and to stream the collected indicator data to the analysis end. The analysis engine module is used to perform time-series analysis on the indicator data based on a machine learning model to predict potential failure risks, and to perform threshold comparison on the indicator data in conjunction with a preset rule base to identify failures that have occurred, thereby generating a failure diagnosis result that includes the failure type and severity level. The decision controller module is used to match the corresponding repair strategy from the multi-level self-healing strategy library according to the fault type and severity level in the fault diagnosis results, and generate execution instructions including node isolation, task migration, node restart or node offline maintenance. The scheduler module is used to respond to the execution command by performing corresponding isolation marking, lossless migration of computing tasks, system reset or resource masking operations on the graphics processing unit node, and verifying the node status after the operation is completed to determine whether to restore the node to the cluster or trigger a manual maintenance notification.

8. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the GPU node automatic fault detection and repair method as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements an automatic fault detection and repair method for GPU nodes as described in any one of claims 1-6.

Images