GPU (Graphics Processing Unit) fault detection and recovery method, equipment and medium

Abstract

The invention discloses a GPU (Graphics Processing Unit) fault detection and recovery method, which relates to the technical field of GPU detection, is used for solving the problem of inaccurate detection in the prior art, and comprises the following steps: collecting operation indexes of a plurality of GPU nodes in a GPU cluster; constructing graph structure data based on the operation indexes; inputting the graph structure data into a trained graph neural network model to obtain a health degree evaluation result corresponding to the GPU node; in response to the health degree evaluation result indicating abnormity, selecting and executing corresponding adaptive recovery measures from the candidate recovery actions according to the severity of the abnormity; and recording an execution effect, and dynamically optimizing a selection strategy of a subsequent recovery measure based on historical execution effect data. The invention further discloses electronic equipment and a computer storage medium. According to the method, multi-dimensional indexes are fused through graph structure data, the GPU health degree is automatically evaluated by using a graph neural network, and hierarchical recovery and strategy learning are triggered based on an evaluation result.

Description

Technical Field

[0001] This invention relates to the field of GPU detection technology, and more particularly to a GPU fault detection and recovery method, device and medium. Background Technology

[0002] With the widespread application of artificial intelligence, deep learning, and high-performance computing, large-scale GPU clusters have become a critical infrastructure for computationally intensive tasks. However, frequent failures of GPU nodes and their high-speed interconnect links can lead to task interruptions, resource waste, and decreased system availability.

[0003] Currently, existing technologies for detecting GPU faults mostly employ threshold-based methods. Typically, static thresholds are set for indicators such as utilization, temperature, or memory errors of a single GPU, and an alarm is triggered when a certain indicator exceeds the threshold.

[0004] These existing technical solutions have the following limitations: Relying on a single metric leads to a high false alarm rate: Focusing only on isolated metrics of a single GPU makes it impossible to distinguish between GPU utilization fluctuations caused by link issues and GPU malfunctions, which easily leads to false alarms.

[0005] Ignoring topology dependencies leads to inaccurate fault localization: Existing methods cannot characterize the complex topological dependencies formed between GPU nodes through high-speed interconnects. When a node exhibits abnormal behavior due to a link problem, the static threshold method cannot trace the root cause of the fault, resulting in ambiguous fault localization and difficulty in matching corresponding recovery measures.

[0006] The recovery strategy is rigid and inefficient: After an anomaly is detected, a fixed recovery process is usually adopted (such as directly restarting the node). There is a lack of an adaptive recovery mechanism that classifies the fault according to its severity and type. Furthermore, the recovery strategy cannot be optimized based on historical experience, resulting in long recovery times and affecting business continuity.

[0007] Therefore, there is a need for an intelligent fault detection and rapid recovery method that can integrate the multi-dimensional operating status of GPU clusters, characterize the dependencies between nodes, and have self-optimization capabilities, in order to improve the overall stability and operational efficiency of large-scale GPU clusters. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, one of the objectives of this invention is to provide a GPU fault detection and recovery method, which integrates multi-dimensional indicators through graph structure data, automatically assesses GPU health using graph neural networks, and triggers graded recovery and strategy learning based on the assessment results.

[0009] One of the objectives of this invention is achieved through the following technical solution: A GPU fault detection and recovery method includes the following steps: Collect operational metrics from multiple GPU nodes in a GPU cluster, including single-card-level metrics and link-level metrics; Based on the aforementioned operational metrics, a graph structure data representing the topology and state of the GPU cluster is constructed. The graph structure data is input into a trained graph neural network model to obtain the health assessment results corresponding to the GPU nodes; In response to an abnormality indicated by the health assessment result, an appropriate adaptive recovery measure is selected from multiple candidate recovery actions and executed according to the severity of the abnormality; Record the execution effect of the adaptive recovery measures, and dynamically optimize the selection strategy of subsequent recovery measures based on historical execution effect data.

[0010] Furthermore, the single-card level metrics include at least one of power, current, temperature, GPU utilization, memory utilization, and memory error rate; the link level metrics include at least one of NVLink bandwidth, latency, InfiniBand throughput, packet loss rate, and retransmission rate. By integrating these two categories of multi-dimensional metrics—single-card internal status and link communication quality—comprehensive data is provided for fault detection. This enables the system to accurately distinguish between GPU card failures and "false" failures caused by link issues, thereby significantly reducing false alarm rates and achieving accurate root cause localization.

[0011] Furthermore, based on the aforementioned operational metrics, a graph structure data characterizing the topology and state of the GPU cluster is constructed, including: The data of the aforementioned operating indicators are preprocessed to eliminate differences in dimensions and instantaneous fluctuations; The processed operational metrics data are combined with the cluster's physical topology to construct a mathematical structure acceptable to a graph neural network.

[0012] Furthermore, in the graph structure data, GPU nodes serve as graph nodes, and communication links between nodes serve as graph edges. Mapping the physical topology of the GPU cluster to graph structure data naturally characterizes the dependencies between nodes, laying a data structure foundation for subsequent use of graph neural networks to discover cross-node, propagating faults caused by communication link degradation, and solving the problem that traditional methods cannot handle topological dependencies.

[0013] Furthermore, the graph neural network model dynamically calculates node aggregation weights based on the edge features constituted by the link-level metrics, reducing the weights of low-quality links and increasing the weights of high-quality links, and its calculation satisfies: , in, For neighboring nodes For nodes Aggregate weights, This represents a linear transformation or multilayer perceptron, used to dynamically adjust weights based on link-level metrics. Represents a node The set of neighboring nodes, This represents a traversal variable used to iterate over nodes. The set of neighboring nodes Each node in the model is considered. By dynamically calculating aggregation weights using edge features, the model can "sense" the link quality during information aggregation. The weights of low-quality links (high error rate, high latency) are automatically reduced, effectively isolating abnormal interference and greatly improving the robustness and accuracy of health assessment.

[0014] Furthermore, the calculation of the health assessment result includes: Obtain the final representation of each node after aggregation of the graph neural network model. ; Based on the final representation of the node, the node health score is calculated using the Sigmoid activation function, and the calculation satisfies: ,in, Represents a node Health score, Indicates the weighting coefficient. This represents the first bias vector; The link health score is obtained by concatenating the health features of the GPU nodes at both ends of the link with the link's own features. Its calculation satisfies the following conditions: ,in, Indicates the health of the link. Representing neighboring nodes For nodes Aggregate weights, This represents the second bias vector. It represents vector concatenation. It provides a unified and quantitative evaluation framework for node and link health, and outputs the health scores of nodes and links simultaneously through a differentiable model. This not only enables fine-grained fault location, but also uses a scientific score calculation method that comprehensively considers the node's own state and the surrounding environment, making the evaluation results more convincing and reliable.

[0015] Furthermore, in response to the health assessment result indicating an anomaly, and based on the severity of the anomaly, a corresponding adaptive recovery measure is selected from multiple candidate recovery actions and executed, including: Based on the health assessment results, the severity of abnormalities is divided into minor abnormalities, moderate abnormalities, and severe abnormalities; Perform a CPU reset for nodes with minor anomalies; For nodes with moderate anomalies, perform a GPU driver or IB network card restart; For severely abnormal nodes, node isolation is implemented and the issue is reported to the cluster management system. A tiered, progressive recovery mechanism is implemented, selecting the appropriate recovery method based on the severity of the fault, thus avoiding service interruptions or fault persistence caused by a mismatch between the fault and the recovery method. This precise matching handling strategy maximizes business continuity while rapidly recovering from faults at minimal cost, achieving the optimal balance between recovery efficiency and business impact.

[0016] Furthermore, the execution effect of the adaptive recovery measures is recorded, and the selection strategy for subsequent recovery measures is dynamically optimized based on historical execution effect data, including: Maintain a statistical success rate based on the number of successful attempts for each type of recovery action; When an anomaly is encountered, recovery actions with a high success rate are prioritized. If the prioritized action fails to eliminate the anomaly, a more intensive recovery action is automatically triggered. This enables the system to learn from historical experience. The success rate-based prioritization and automatic upgrade mechanism allow the system's recovery strategy to continuously optimize over time, gradually adapting to the hardware characteristics and workload of the specific cluster.

[0017] A second objective of this invention is to provide an electronic device that performs one of the objectives of the invention, comprising a processor, a storage medium, and a computer program, wherein the computer program is stored in the storage medium and, when executed by the processor, implements the aforementioned GPU fault detection and recovery method.

[0018] A third objective of this invention is to provide a computer-readable storage medium that stores one of the objectives of the invention, having stored thereon a computer program that, when executed by a processor, implements the aforementioned GPU fault detection and recovery method.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention integrates multi-dimensional monitoring, intelligent diagnosis, hierarchical handling, and strategy optimization into a seamless automated process, changing the traditional fragmented operation and maintenance model that relies on manual intervention and realizing the autonomy and self-healing of GPU clusters. By using graph neural networks to perform holistic analysis of cluster status, it significantly improves the accuracy of fault location and can identify potential risks, avoiding the inaccuracies caused by fault detection using a single static threshold. Through a hierarchical recovery mechanism, it restores faults at the lowest cost and fastest speed, maximizing business continuity and reducing task interruption time. Attached Figure Description

[0020] Figure 1 This is a flowchart of the GPU fault detection and recovery method in Example 1; Figure 2 This is a schematic diagram of the anomaly detection module structure in Embodiment 2; Figure 3 This is a structural block diagram of the electronic device in Embodiment 4. Detailed Implementation

[0021] The present invention will now be described in more detail with reference to the accompanying drawings. It should be noted that the following description of the present invention with reference to the accompanying drawings is merely illustrative and not restrictive. Various embodiments can be combined with each other to form other embodiments not shown in the following description.

[0022] Example 1 Example 1 provides a GPU fault detection and recovery method. It aims to abstract the GPU cluster as a graph structure, utilize graph neural networks to fuse multi-dimensional indicators for intelligent health assessment, and accurately locate faults. Subsequently, the system automatically executes tiered recovery measures based on the severity of the fault and dynamically optimizes the recovery strategy based on historical performance data. Ultimately, a fully automated closed-loop self-healing system is achieved, from detection and decision-making to recovery and optimization, significantly improving cluster stability and operational efficiency.

[0023] This embodiment's method collects GPU single-card level and link-level metrics in real time and constructs graph structure data containing node and edge features. It then uses a graph neural network (GNN) to model this graph structure data, calculates a health score, and achieves accurate identification of GPU faults and potential risks. Based on the fault severity, it executes tiered adaptive recovery measures, including GPU reset, driver or system restart, and node isolation, and dynamically optimizes action priorities based on historical feedback using a policy learning module. Compared to traditional methods relying on single metrics or static thresholds, this method reduces false alarm rates, improves fault location accuracy, and enables rapid recovery and self-healing optimization in large-scale GPU clusters, thereby significantly improving system stability and computational efficiency.

[0024] To enable real-time monitoring of GPU node and link status, accurate anomaly identification, and triggering tiered adaptive recovery measures, action priority optimization is achieved through policy learning. Please refer to [link / reference needed]. Figure 1 As shown, this embodiment provides a GPU fault detection and recovery method, including the following steps: S1. Collect the operating metrics of multiple GPU nodes in the GPU cluster, including single-card level metrics and link level metrics; The single-card level metrics in S1 include at least one of power, current, temperature, GPU utilization, memory utilization, and memory error rate; the link level metrics include at least one of NVLink bandwidth, latency, InfiniBand throughput, packet loss rate, and retransmission rate.

[0025] Data acquisition for S1 can be accomplished by a data acquisition agent deployed on GPU nodes. This agent runs as a daemon on each compute node, periodically calling the underlying interface to obtain GPU running status and link communication information. Single-card level metrics are preferably obtained through the NVIDIA NVML API or the nvidia-smi tool; link level metrics can be obtained using commands such as nvidia-smi nvlink -s, ibstat, and ibqueryerrors.

[0026] The collection period can be dynamically adjusted according to the cluster load, usually set between 1s and 10s. The agent caches the data of the most recent N time windows locally and uses the moving average or exponential weighting method to smooth the instantaneous spikes. At the same time, it uses missing value interpolation or previous time step filling method to solve the problem of abnormal missing values.

[0027] S2. Based on the aforementioned operational metrics, construct graph-structured data representing the topology and state of the GPU cluster; The construction of the S2 graph structure data includes: The data of the aforementioned operating indicators are preprocessed to eliminate differences in dimensions and instantaneous fluctuations; The processed operational metrics data are combined with the cluster's physical topology to construct a mathematical structure acceptable to a graph neural network.

[0028] The elimination of dimensional differences and instantaneous fluctuations can be achieved through data processing methods such as normalization and moving averages. These methods are mature existing technologies, therefore, this embodiment will not elaborate on their specific processing procedures.

[0029] In graph-structured data, GPU nodes are treated as graph nodes, and the communication links between nodes are treated as graph edges. Specifically, the node feature vector consists of multi-dimensional indicators for each GPU, representing the individual GPU's health status; the edge feature matrix consists of link performance parameters, representing the communication relationships between GPUs; a time-series graph of indicators, G=(V,E,X,A), is constructed, where V is the set of nodes, E is the set of edges, X is the node feature, and A is the edge feature. Through indicator fusion, both individual GPU health and cross-node communication status can be reflected simultaneously, providing a data foundation for subsequent anomaly detection.

[0030] S3. Input the graph structure data into the trained graph neural network model to obtain the health assessment result corresponding to the GPU node; The graph neural network model (GNN) in S3 uses the edge features formed by the link-level metrics and dynamically calculates the node aggregation weights based on the edge features, thereby reducing the weights of low-quality links and increasing the weights of high-quality links.

[0031] Specifically, node feature vectors Composed of single GPU metrics, such as power consumption, current consumption, temperature, memory usage, memory error rate, GPU utilization, core frequency, etc.; edge feature vectors. It consists of link metrics, such as NVLink bandwidth, latency, and error rate, and InfiniBand throughput, packet loss rate, and retransmission rate; the graph structure is G=(V,E,X,A): where V represents the set of GPU nodes, E represents the set of links, X represents the node feature matrix, and A represents the edge feature matrix.

[0032] In this embodiment, the GNN uses an improved Graph Convolutional Network (GCN) as its base model. For the l-th layer, the nodes... The formula for updating the representation vector is: ,in, Represents a node The set of neighboring nodes; For nodes The feature representation at layer l is initialized with the input features. ; The learnable weight matrix is ​​the sum of the model's global parameters. Real-time weight parameters of the link; weight parameter updates depend on loss feedback from global training data; matrix. The dimension is determined by the input feature dimension of the l-th layer. and output feature dimension A joint decision, ultimately presented as The matrix form, This refers to the length of the feature vector of the node in layer l, corresponding to the original / intermediate feature dimension of the node in that layer (e.g., when including metrics such as GPU power, temperature, and link bandwidth). That is, the number of indicators). The target length of the node feature vector after the l-th layer transformation is set by the requirements of feature representation capability during model design. The Sigmoid is a non-linear activation function. For neighboring nodes For nodes The aggregate weights, whose parameter calculations depend on the real-time characteristics of individual links, aim to accurately reflect the current information transmission value of the links, derived from edge feature vectors. The calculation yielded: , in, For neighboring nodes For nodes Aggregate weights, This represents a linear transformation or multilayer perceptron, used to dynamically adjust weights based on link-level metrics. It solves the problem that traditional static weights cannot adapt to the complex topologies of GPU clusters, specifically in the following scenarios: Weight reduction for low-quality links: When a link (such as NVLink) experiences performance anomalies (high error rate, high packet loss rate, high latency), its link feature vector... The corresponding abnormal indicator values ​​will change significantly. By capturing and calculating these abnormal indicators, a lower function value is output. After exp and normalization, the aggregate weight corresponding to this link is determined. This will decrease accordingly, preventing "bad data" from abnormal links from interfering with the feature aggregation of healthy nodes and reducing the risk of misjudgment. High-quality link weights are increased: when link performance is stable (sufficient bandwidth, low error rate, no packet loss), All indicators are within the normal range. A higher output function value results in a higher aggregate weight for that link. Increasing the weight of healthy nodes allows them to more fully utilize neighbor information from high-quality links, improving the accuracy of health score calculations. This aligns with the actual operational characteristics of GPU clusters, where "high-quality links dominate inter-node communication." It also results in lower weights for connections with poor link quality (such as those with high NVLink error rates) during aggregation, while high-quality links receive higher weights, thus better reflecting the actual operational characteristics of GPU topologies. Represents a node The set of neighboring nodes, that is, all nodes that are related to the node A set consisting of directly connected nodes. This represents a traversal variable used to iterate over nodes. The set of neighboring nodes Each node in the process.

[0033] The node health score is output through a fully connected layer and a sigmoid activation function. The calculation of the health assessment result in S3 includes: Obtain the final representation of each node after aggregation of the graph neural network model. ; Based on the final representation of the node, the node health score is calculated using the Sigmoid activation function, and the calculation satisfies: ,in, Represents a node Health score, It includes various characteristics of the node itself and its neighboring nodes (such as GPU power, temperature, link bandwidth, and other metrics). This represents the health score of the GPU node; the lower the value, the greater the risk. These are the weight coefficients (training parameters). This can be understood as a set of "weighting coefficients," which assign different levels of importance to different features. For example, in GPU fault detection, temperature might be a more important feature when determining the health status of a node. The corresponding temperature feature will have a larger weight, thus increasing the impact of temperature features on node health scores in subsequent calculations. The impact is therefore more significant; This represents the first bias vector, used to adjust the result after the linear transformation, making the model fit better. Even when the weighted sum of all features is 0, the bias... It also allows the model to output a basic numerical value, preventing the model from only being able to output a fixed value due to a feature weighted sum of 0 (for example, the Sigmoid function outputs 0.5 when the input is 0). It provides the model with an "offset," allowing the model to better adapt to the distribution of the data, further improving the accuracy of calculating the health score of GPU nodes, and making nodes with different health states more clearly distinguishable.

[0034] The link health score is obtained by concatenating the health features of the GPU nodes at both ends of the link with the link's own features. Its calculation satisfies the following conditions: ,in, Indicates the health of the link. Representing neighboring nodes For nodes Aggregate weights, represents the second bias vector, and || represents vector concatenation.

[0035] S4. In response to the health assessment result indicating an abnormality, select and execute corresponding adaptive recovery measures from multiple candidate recovery actions according to the severity of the abnormality; S4 specifically includes: Based on the health assessment results, the severity of abnormalities is divided into minor abnormalities, moderate abnormalities, and severe abnormalities; For nodes with minor anomalies, perform a CPU reset; additionally, if the GPU is running a low-priority task, reset after migration; check the recovery status, and terminate the process if the health status improves. For nodes with moderate anomalies, perform a GPU driver or IB network card restart; in addition, if the problem persists, perform a node operating system restart; and automatically trigger task migration to ensure business continuity. For severely abnormal nodes, perform node isolation, mark the GPU or the entire machine as unschedulable, report to the cluster management system, block the node, and push an alarm to the operation and maintenance platform for manual investigation.

[0036] The anomaly detection in this embodiment is based on a preset threshold. If Determine the GPU node Abnormal; if Determine the link ( Anomalies. Since GNNs inherently possess the ability to propagate neighbor information, if a node malfunctions, its impact on neighboring nodes will be reflected in feature aggregation. Therefore, the method in this embodiment can not only detect local single-point failures, but also identify potential cross-node risks caused by link anomalies, thereby improving the foresight of the detection.

[0037] S5. Record the execution effect of the adaptive recovery measures, and dynamically optimize the selection strategy of subsequent recovery measures based on historical execution effect data.

[0038] S5 includes: For each type of recovery action, maintain a statistical success rate based on the number of successful attempts. The statistics include the number of successful system maintenance attempts (number of times the GPU recovered after the action was executed, total number of attempts, average recovery time, and degree of impact on task execution) for each type of processing action (GPU Reset, driver / node restart, isolation).

[0039] The success rate is updated in real time after each action is executed. .

[0040] When an anomaly is encountered, recovery actions with a high success rate are prioritized; if the prioritized action fails to eliminate the anomaly, a recovery action with a higher intervention intensity is automatically triggered.

[0041] Specifically, when encountering new GPU anomalies, the system prioritizes actions with high historical success rates. If an action has failed repeatedly recently, it automatically escalates to a stronger handling measure (such as upgrading from Reset to reboot). As runtime increases and statistical data becomes richer, the system develops a better action priority ranking without human intervention. This optimization process does not require complex reinforcement learning algorithms; it can achieve long-term experience-driven optimization of processing strategies based solely on statistical feedback, thereby reducing implementation complexity while ensuring stability.

[0042] In this embodiment, mild anomalies are preferably resolved by resetting the GPU using the NVML API command `nvidia-smi --gpu-reset -i gpu x`. Moderate anomalies are resolved by restarting the GPU driver or InfiniBand driver using `systemctl restart nvidia-driver` or `systemctl restart openibd`. Severe anomalies are resolved by calling the cluster scheduler interface to mark the node as unavailable and performing task migration or isolation. The system also maintains an action effect database, recording the execution and success rates of each type of recovery action, and defines a success rate formula to dynamically adjust action priorities. When a certain type of operation maintains a high success rate for a long period, it is executed first. If it fails continuously, it is automatically upgraded to a stronger recovery measure, thereby achieving a closed loop of detection, handling, and strategy optimization.

[0043] Example 2 Example 2 is a further explanation of anomaly detection using a graph neural network model.

[0044] Please refer to Figure 2 The diagram shown corresponds to the anomaly detection module structure described in Embodiment 1. This embodiment is a refinement of anomaly detection based on GCN. After receiving standardized indicators, the central monitoring service calls the graph construction module to generate a graph structure G=(V,E,X,A), where V is the GPU node set, E is the link set, X is the node feature matrix, and A is the edge feature matrix. Node features include power, current, temperature, memory usage, error rate, etc., while edge features include NVLink bandwidth, latency, error rate, and InfiniBand packet loss rate, retransmission rate, etc. This graph is fed into the anomaly detection module as input. This module is designed based on Graph Convolutional Network (GCN) and includes an input layer, two graph convolutional layers, and an output layer. The input layer receives features. The first convolutional layer aggregates the features of nodes and their neighbors according to edge weights. The edge weights are calculated by MLP from the edge features, which can dynamically reflect the link quality. When the error rate of a link increases, its weight in the aggregation will decrease, reducing the interference of abnormal links on the health calculation. The second convolutional layer continues to propagate information over a wider range, enabling potential risks across nodes to be captured. The final node representation is fed into the fully connected layer, and a health score is output through the Sigmoid activation function. A lower value indicates a higher risk. The link health score is calculated by concatenating the edge features and the features of the two endpoints and inputting the result into the MLP. When the GPU node score is below the threshold θ, it is considered a fault risk, and the same applies to the link. Due to the multi-layer propagation mechanism of GCN, abnormal states will spread among neighbors, thus not only detecting single-point failures but also identifying cross-node risks caused by link problems in advance. For example, when the NVLink link error rate of a GPU continues to rise, this information will be captured by neighboring nodes during convolutional aggregation, causing a decrease in neighbor health, thereby achieving proactive early warning.

[0045] Example 3 Example 3 is a further explanation of the subsequent recovery measures for dynamic optimization.

[0046] For the recovery process following GPU fault detection, this embodiment introduces an adaptive strategy optimization mechanism based on statistical feedback. Specifically, fault recovery actions are divided into three categories: GPU reset, driver or system restart, and node isolation. During system operation, the recovery module records the execution information of each recovery operation, including: action type, execution time, recovery duration, task impact, and final result (success or failure). This information is written to the action effect database for subsequent statistical analysis. The system maintains a success rate metric for each type of action, defined as: Simultaneously, the system maintains average recovery time and task impact as auxiliary indicators. After each action is executed, the database updates these statistics in real time. For example, if a GPU reset action succeeds 82 out of the last 100 attempts, the success rate is 82%. When encountering a new GPU anomaly, the scheduling module prioritizes the action with the highest success rate. If this action fails more than a preset number of times (e.g., 3 times), it automatically escalates to the next level of recovery measures. For severe anomalies (such as prolonged link failure or hardware damage), the system directly isolates the node and triggers task migration to avoid business interruption. As the cluster runtime increases, statistical data accumulates, and the action priority ranking is dynamically optimized. In a certain HPC cluster environment, the long-term success rate of GPU Reset remains above 90%. Therefore, when the system encounters similar anomalies in the future, it will automatically prioritize Reset operations, thereby reducing the probability of task interruption and recovery latency. In another environment, due to the aging of some hardware causing a decrease in the Reset success rate, the system will be more inclined to directly execute driver restart or node isolation. The optimization process of this strategy does not require the introduction of a complex reinforcement learning framework. It can achieve long-term experience-driven decision optimization by relying solely on statistical feedback, ensuring low implementation complexity, strong maintainability, and balancing system stability and efficiency.

[0047] Example 4 Figure 3This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention, as shown below. Figure 3 As shown, the electronic device includes a processor 410, a memory 420, an input device 430, and an output device 440; the number of processors 410 in the computer device can be one or more. Figure 3 Taking a processor 410 as an example; the processor 410, memory 420, input device 430, and output device 440 in the electronic device can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0048] The memory 420, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules. The processor 410 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 420, thereby implementing the GPU fault detection and recovery methods of Embodiments 1 to 3 described above.

[0049] The memory 420 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 420 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 420 may further include memory remotely located relative to the processor 410, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0050] The input device 430 can be used to receive input user identity information, node data, and detection data, etc. The output device 440 may include a display screen or other display device.

[0051] Example 5 Embodiment 5 of the present invention also provides a storage medium containing computer-executable instructions, which can be used by a computer to execute a GPU fault detection and recovery method, the method comprising: Collect operational metrics from multiple GPU nodes in a GPU cluster, including single-card-level metrics and link-level metrics; Based on the aforementioned operational metrics, a graph structure data representing the topology and state of the GPU cluster is constructed. The graph structure data is input into a trained graph neural network model to obtain the health assessment results corresponding to the GPU nodes; In response to an abnormality indicated by the health assessment result, an appropriate adaptive recovery measure is selected from multiple candidate recovery actions and executed according to the severity of the abnormality; The effectiveness of the adaptive recovery measures is recorded, and the selection strategy for subsequent recovery measures is dynamically optimized based on historical performance data.

[0052] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause an electronic device (which may be a mobile phone, personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0053] For those skilled in the art, various other corresponding changes and modifications can be made based on the technical solutions and concepts described above, and all such changes and modifications should fall within the protection scope of the claims of this invention.

Claims

1. A GPU fault detection and recovery method, characterized in that, Includes the following steps: Collect operational metrics from multiple GPU nodes in a GPU cluster, including single-card-level metrics and link-level metrics; Based on the aforementioned operational metrics, a graph structure data representing the topology and state of the GPU cluster is constructed. The graph structure data is input into a trained graph neural network model to obtain the health assessment results corresponding to the GPU nodes; In response to an abnormality indicated by the health assessment result, an appropriate adaptive recovery measure is selected from multiple candidate recovery actions and executed according to the severity of the abnormality; Record the execution effect of the adaptive recovery measures, and dynamically optimize the selection strategy of subsequent recovery measures based on historical execution effect data.

2. The GPU fault detection and recovery method as described in claim 1, characterized in that, The single-card level metrics include at least one of power, current, temperature, GPU utilization, memory utilization, and memory error rate; the link level metrics include at least one of NVLink bandwidth, latency, InfiniBand throughput, packet loss rate, and retransmission rate.

3. The GPU fault detection and recovery method as described in claim 1, characterized in that, Based on the aforementioned operational metrics, a graph-structured data representing the topology and state of the GPU cluster is constructed, including: The data of the aforementioned operating indicators are preprocessed to eliminate differences in dimensions and instantaneous fluctuations; The processed operational metrics data are combined with the cluster's physical topology to construct a mathematical structure acceptable to a graph neural network.

4. The GPU fault detection and recovery method as described in claim 1 or 3, characterized in that, In the graph structure data, GPU nodes serve as graph nodes, and the communication links between nodes serve as graph edges.

5. The GPU fault detection and recovery method as described in claim 1, characterized in that, The graph neural network model dynamically calculates node aggregation weights based on the edge features formed by the link-level metrics, reducing the weights of low-quality links and increasing the weights of high-quality links. Its calculation satisfies the following: ， in, For neighboring nodes For nodes Aggregate weights, This represents a linear transformation or multilayer perceptron, used to dynamically adjust weights based on link-level metrics. Represents a node The set of neighboring nodes, This indicates the variable to be iterated over.

6. The GPU fault detection and recovery method as described in claim 1, characterized in that, The calculation of the health assessment results includes: Obtain the final representation of each node after aggregation of the graph neural network model. ; Based on the final representation of the node, the node health score is calculated using the Sigmoid activation function, and the calculation satisfies: ,in, Represents a node Health score, These are the weighting coefficients. This represents the first bias vector; The link health score is obtained by concatenating the health features of the GPU nodes at both ends of the link with the link's own features. Its calculation satisfies the following conditions: ,in, Indicates the health of the link. Representing neighboring nodes For nodes Aggregate weights, represents the second bias vector, and || represents vector concatenation.

7. The GPU fault detection and recovery method as described in claim 1 or 6, characterized in that, In response to an abnormality indicated by the health assessment result, and based on the severity of the abnormality, an adaptive recovery measure is selected from multiple candidate recovery actions and executed, including: Based on the health assessment results, the severity of abnormalities is divided into minor abnormalities, moderate abnormalities, and severe abnormalities; Perform a CPU reset for nodes with minor anomalies; For nodes with moderate anomalies, perform a GPU driver or IB network card restart; For nodes with severe anomalies, perform node isolation and report to the cluster management system.

8. The GPU fault detection and recovery method as described in claim 1, characterized in that, Record the execution effect of the adaptive recovery measures, and dynamically optimize the selection strategy of subsequent recovery measures based on historical execution effect data, including: Maintain a statistical success rate based on the number of successful attempts for each type of recovery action; When an anomaly is encountered, recovery actions with a high success rate are prioritized; if the prioritized action fails to eliminate the anomaly, a recovery action with a higher intervention intensity is automatically triggered.

9. An electronic device comprising a processor, a storage medium, and a computer program, wherein the computer program is stored in the storage medium, characterized in that, When the computer program is executed by the processor, it implements the GPU fault detection and recovery method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the GPU fault detection and recovery method according to any one of claims 1 to 8.

Images