GPU graphics processor troubleshooting methods and systems

Abstract

The GPU graphics processor fault repair method and system provided in this application first determines the connectivity fault tolerance threshold based on the node communication topology of the currently running training service. This threshold represents the actual node communication dependencies of the training service. Then, it enters the fault batch division stage. Using the connectivity fault tolerance threshold as a constraint, the batch division is completed in combination with the error information reported by the faulty nodes. This ensures that the repair scale of each batch does not exceed the safety limit that the service can bear, avoiding the risk of service communication interruption and training task crash caused by repair actions. At the same time, it can maximize the number of repaired nodes in a single batch within the safety boundary. This solves the problems of extremely low efficiency of single-node rolling repair and lack of safety guarantee of experience-based batch repair in the existing technology. Batch fault repair can be completed without interrupting the training task during the GPU repair process.

Description

Technical Field

[0001] This application belongs to the field of GPU fault repair technology, and in particular relates to a fault repair method and system for a GPU graphics processor. Background Technology

[0002] With the exponential growth in the number of training parameters and computational load of large-scale models, large-scale GPU clusters have become the core infrastructure for AI training. To ensure gradient sharing, large-scale GPU clusters require communication dependencies between nodes. However, GPU nodes operating at full load for extended periods are prone to various recoverable faults. If these communication dependencies are disrupted during repair, training must be paused. Existing repair solutions, such as single-node rolling repair and batch repair, cannot simultaneously balance repair efficiency and business continuity, easily leading to business communication interruptions and training task crashes. Currently, it is impossible to achieve efficient fault repair without business interruption. Summary of the Invention

[0003] This application provides a method and system for troubleshooting GPU graphics processors, in order to solve the problem that it is currently impossible to achieve efficient troubleshooting without service interruption.

[0004] This application provides a fault repair method for GPU graphics processing units, applied to a distributed training cluster containing multiple GPU nodes, wherein each GPU node communicates with at least two GPU nodes to form a communication topology. The fault repair method includes: Based on the communication topology graph, a connectivity fault tolerance threshold is determined. The connectivity fault tolerance threshold is the maximum number of GPU nodes that can be repaired simultaneously in the same batch while keeping the current training service communication topology connectivity value higher than a set value. The communication topology connectivity value is the minimum value among the maximum number of independent communication paths corresponding to all node pairs in the communication topology graph. The set value is greater than or equal to zero. Based on the error information reported by the repairable faulty GPU nodes and the connectivity fault tolerance threshold, all repairable faulty GPU nodes are divided into batches to generate a set of faulty nodes corresponding to each repair batch. The corresponding set of faulty nodes will be repaired sequentially according to the generated repair batches.

[0005] Optionally, the steps for establishing the communication topology map include: Collect process execution information of distributed training services, as well as the necessary communication dependencies between nodes; Using the GPU nodes participating in the current training process as the basic units of the graph, and the necessary communication connections between GPU nodes for the training process as the connection relationships, a communication topology graph that perfectly matches the communication logic of the current training process is generated.

[0006] Optionally, the step of determining the communication topology connectivity value includes: Traverse all pairs of GPU nodes in the communication topology graph to generate a set of node pairs to be verified. For each node pair in the set of node pairs, determine the maximum number of independent communication paths between them that do not share any intermediate GPU nodes; The minimum value among the maximum number of independent communication paths corresponding to all node pairs is taken as the communication topology connectivity value.

[0007] Optionally, determining the connectivity fault tolerance threshold based on the communication topology graph includes: Obtain the set values ​​of the communication topology connectivity required for the normal operation of the current training service; The maximum number of GPU nodes that can temporarily detach from the communication topology map in a single batch is calculated when the communication topology connectivity value is continuously higher than a set value. This maximum number is then determined as the connectivity fault tolerance threshold for the current training service.

[0008] Optionally, all repairable faulty GPU nodes are divided into batches to generate a set of faulty nodes corresponding to each repair batch, including: Based on the error information reported by the repairable faulty GPU nodes, determine the fault risk level corresponding to each faulty node; Sort all repairable faulty GPU nodes in descending order of fault risk level; With the constraint that the number of faulty nodes included in a single batch does not exceed the connectivity fault tolerance threshold of the current training service, faulty nodes are sequentially assigned to the corresponding repair batches according to the sorting results, generating a set of faulty nodes corresponding to each repair batch.

[0009] Optionally, before performing fault repair according to the generated repair batch, the following steps are included: Lock the computation execution timing and communication interaction rules for the current training service; For the set of faulty nodes currently awaiting repair, verify that after removing the corresponding node, the connectivity of the communication topology formed by the remaining healthy GPU nodes is still higher than the set value, confirming that all necessary communication paths between the remaining healthy nodes remain available.

[0010] Optionally, after completing the repair operation for a single batch of faulty nodes, the following steps are included: Perform hardware status verification to collect data and confirm that the fault of the GPU node has been cleared after repair, and / or the hardware performance has been restored to the rated standard; Perform a topology connectivity check to confirm that after the repair node reconnects, the communication topology connectivity value of the current training service is higher than the set value; Perform a business operation verification to confirm that the computation and communication processes of the training business are normal after the repair node is connected. After all verifications pass, restore the training business participation permissions of the corresponding GPU node.

[0011] Optionally, retrieve error information reported by the repairable faulty GPU node, including: The fault error codes and hardware operating status data reported by each GPU node are collected in real time through the GPU management interface. Based on a pre-defined library of repairable fault types, the collected error information is assessed for repairability, and GPU nodes that meet the repairability criteria are selected. The library of repairable fault types includes fault types and error codes corresponding to correctable memory errors, driver malfunctions, and firmware status anomalies.

[0012] Optionally, fault repair can be performed on the corresponding set of faulty nodes sequentially according to the generated repair batches, including: Get the set of fault nodes corresponding to the current repair batch, and the fault type corresponding to each fault node; Based on the fault type, a hierarchical repair operation sequence with invasiveness ranging from low to high is matched, and the corresponding repair operation is executed to clear the node fault. Confirm that the repair operations for all faulty nodes in the current batch have been completed, and output the batch repair results.

[0013] This application also provides a GPU graphics processor fault repair system, applied to a distributed training cluster containing multiple GPU nodes, wherein each GPU node communicates with at least two GPU nodes to form a communication topology. The fault repair system includes: The determination module determines a connectivity fault tolerance threshold based on the communication topology graph. The connectivity fault tolerance threshold is the maximum number of GPU nodes that can be repaired simultaneously in the same batch while keeping the current training service communication topology connectivity value higher than a set value. The communication topology connectivity value is the minimum value among the maximum number of independent communication paths corresponding to all node pairs in the communication topology graph, and the set value is greater than or equal to zero. The batch partitioning module divides all repairable faulty GPU nodes into batches based on the error information reported by the repairable faulty GPU nodes and the connectivity fault tolerance threshold, generating a set of faulty nodes corresponding to each repair batch. The repair module repairs the corresponding set of faulty nodes sequentially according to the generated repair batches.

[0014] As can be seen from the above technical solution, the GPU graphics processor fault repair method and system provided in this application first determines the connectivity fault tolerance threshold based on the node communication topology of the currently running training service. This threshold represents the actual node communication dependency relationship of the training service. Then, it enters the fault batch division stage. With the connectivity fault tolerance threshold as a constraint, the batch division is completed in combination with the error information reported by the faulty nodes. This ensures that the repair scale of each batch will not break the safety limit that the service can bear, avoids the risk of service communication interruption and training task crash caused by repair actions, and maximizes the number of repair nodes in a single batch within the safety boundary. This solves the problems of extremely low efficiency of single-node rolling repair and lack of safety guarantee of experience-based batch repair in the existing technology. Batch fault repair can be completed without interrupting the training task during the GPU repair process. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A schematic flowchart of a fault repair method for a GPU graphics processor according to an embodiment of this application is shown; Figure 2 This paper illustrates a schematic diagram of the principle of a first repair batch, as shown in a specific example of an embodiment of this application. Figure 3 This paper illustrates a schematic diagram of the principle of a second repair batch, representing a specific example in an embodiment of this application. Figure 4 This paper illustrates a schematic diagram of the principle of a third repair batch, based on a specific example from an embodiment of this application. Figure 5 A schematic diagram of a fault repair system for a GPU graphics processor is shown in an embodiment of this application. Detailed Implementation

[0017] The rapid development of distributed AI training technology has led to a continuous increase in the demand for computing resources for training large-parameter AI models. A single GPU server can no longer meet the computing power and memory requirements for pre-training and fine-tuning of models with hundreds of billions or more parameters. Therefore, such training tasks are usually deployed in distributed training clusters consisting of tens to thousands of GPU nodes, completing the training process through multi-node collaborative parallel computing. The training cycle of large-parameter AI models is usually several weeks or even months. Throughout the training cycle, the GPU nodes in the cluster need to run at full load continuously. Prolonged full-load operation can cause various hardware, firmware, and driver-level anomalies in the GPU nodes. Most of these anomalies are repairable faults that can be cleared through maintenance operations without replacing hardware. These repairable faults include, but are not limited to, excessive accumulation of correctable errors in memory, abnormal GPU driver operation, abnormal firmware status, and disordered hardware register status.

[0018] Taking a cluster of 100 GPU nodes (with a 10x10 node numbering system) as an example, during training, the operation and maintenance system detected that 20 GPU nodes in the cluster had exceeded the cumulative correctable memory errors. This fault is a repairable fault, but if it is not handled in time, the memory errors will escalate to uncorrectable errors, leading to GPU node offline and training task crashes. The 20 faulty GPU nodes are numbered as follows in the 10x10 node numbering system: (0,0), (0,5), (1,3), (2,7), (3,2), (3,8), (4,1), (4,6), (5,4), (5,9), (6,0), (6,5), (7,3), (7,7), (8,2), (8,8), (9,1), (9,4), (9,6), (9,9). The first value of the node number is the row number, and the second value is the column number, both ranging from 0 to 9.

[0019] For the 20 repairable faulty GPU nodes in the aforementioned 100-node cluster, the existing passive shutdown repair solution executes as follows: First, a stop command is sent to the management node of the training task to terminate the currently running pre-training task, triggering the saving operation of the training state snapshot. Information such as the current model parameters, optimizer status, iteration steps, and learning rate scheduling status are written to persistent storage. Then, all 20 faulty GPU nodes are taken offline from the cluster, and a batch repair operation is performed. After the repair is completed, all nodes are reconnected to the cluster, a restart command is sent to the management node, the pre-training task is started, the saved training state snapshot is read from persistent storage, and the training process is restored.

[0020] Those skilled in the art will understand that the entire process from task stoppage to resumption of operation involves multiple stages, including training task termination, state snapshot saving, node repair, task restart, and state recovery. This will result in a loss of training iteration progress for the corresponding rounds. If multiple similar batch failures occur within the training cycle, the accumulated downtime will extend the overall training cycle. Furthermore, the restart and state recovery processes for large-parameter pre-training tasks carry corresponding risks of failure. If the state snapshot file is corrupted or anomalies such as memory overflow or parameter mismatch occur during the recovery process, all completed training progress will be lost, resulting in irreparable losses of computing resources and time costs.

[0021] For example, regarding the 20 faulty GPU nodes in the aforementioned 100-node cluster, the existing single-node rolling repair solution's execution flow is as follows: the operation and maintenance system sorts the 20 faulty nodes from highest to lowest based on the cumulative number of memory errors, and then repairs each faulty node one by one according to the sorting result. Specifically, the complete repair process for a single node is as follows: first, the target faulty node is temporarily taken offline from the training cluster, and the training task load borne by the node is migrated to other healthy nodes. Then, the repair operation is performed on the node. After the repair is completed, the node is reconnected to the cluster, its training task load is restored, and after confirming that the node is normally participating in the training process, the repair process for the next faulty node is started, and so on, until all 20 faulty nodes are repaired.

[0022] It should be noted that the entire process of taking a single node offline, migrating its load, repairing, reconnecting, and restoring its load takes a fixed amount of time. The total repair time for 20 nodes is 20 times that of a single node. Compared to the shutdown repair solution, this solution does not require stopping the overall training task and will not cause any loss of training progress. However, since only one node can be repaired at a time, the overall repair efficiency is low, and the cluster remains in a state of compromised computing power throughout the entire repair cycle. The total computing power of the 20 faulty nodes accounts for 20% of the total computing power of the cluster. During the repair cycle, the total available computing power of the cluster is only 80% of the normal state, which will lead to a decrease in training iteration speed. At the same time, the frequent offline and reconnection of a single node will continuously interfere with the gradient synchronization and computation timing processes of the training task, which can easily cause gradient jitter and a decrease in convergence speed during training. If the gradient jitter lasts for a long time, it will increase the total number of iterations required for model convergence, further extending the overall training cycle.

[0023] It can be seen that none of the above methods can achieve efficient fault repair without service interruption.

[0024] In this application embodiment, in order to solve the problems existing in the prior art, a fault repair method for GPU graphics processor is provided. The method is applied to a distributed training cluster containing multiple GPU nodes, wherein each GPU node communicates with at least two GPU nodes to form a communication topology.

[0025] It should be noted that the method in this application embodiment is based entirely on the aforementioned distributed training cluster scenario with 100 GPU nodes. All technical details are described in accordance with the specific implementation process in this scenario to ensure sufficient disclosure of the technical solution. However, it is understood that the 100 GPU nodes listed in this application embodiment are exemplary, and the core concept of this application can also be implemented with 200, 150, and 80 GPU nodes in other embodiments. This application does not limit the number of nodes used to implement this application, and such examples are not listed here.

[0026] like Figure 1 As shown, this application provides a fault repair method for GPU graphics processing units, applied to a distributed training cluster containing multiple GPU nodes, wherein each GPU node communicates with at least two GPU nodes to form a communication topology. The fault repair method includes: S1: Determine a connectivity fault tolerance threshold based on the communication topology graph. The connectivity fault tolerance threshold is the maximum number of GPU nodes that can be repaired simultaneously in the same batch while keeping the current training service communication topology connectivity value higher than a set value. The communication topology connectivity value is the minimum value among the maximum number of independent communication paths corresponding to all node pairs in the communication topology graph. The set value is greater than or equal to zero. S2: Based on the error information reported by the repairable faulty GPU nodes and the connectivity fault tolerance threshold, all repairable faulty GPU nodes are divided into batches to generate a set of faulty nodes corresponding to each repair batch. S3: Repair the corresponding set of faulty nodes in sequence according to the generated repair batch.

[0027] The GPU graphics processor fault repair method provided in this application first determines a connectivity fault tolerance threshold based on the node communication topology of the currently running training service. This threshold represents the actual node communication dependencies of the training service. Then, it enters the fault batch division stage. Using the connectivity fault tolerance threshold as a constraint, the batch division is completed in combination with the error information reported by the faulty nodes. This ensures that the repair scale of each batch does not exceed the safety limit that the service can bear, avoiding the risk of service communication interruption and training task crash caused by repair actions. At the same time, it can maximize the number of repaired nodes in a single batch within the safety boundary. This solves the problems of extremely low efficiency of single-node rolling repair and lack of safety guarantee of experience-based batch repair in the existing technology. Batch fault repair can be completed without interrupting the training task during the GPU repair process.

[0028] It should be noted that the normal operation of distributed training tasks depends on the collaborative communication between GPU nodes. Different parallel training modes will generate different communication dependencies between nodes. For example, in data parallel mode, there is a bidirectional communication dependency for gradient synchronization between nodes; in tensor parallel mode, there is a bidirectional communication dependency for the transmission of intermediate computation results between nodes; and in pipeline parallel mode, there is a unidirectional communication dependency for the connection of computation stages between nodes. These communication dependencies together constitute the complete communication logic of the training business, thus forming the communication topology diagram in the embodiments of this application.

[0029] In this embodiment of the application, the step of establishing the communication topology map includes: Collect process execution information of distributed training services, as well as the necessary communication dependencies between nodes; Using the GPU nodes participating in the current training process as the basic units of the graph, and the necessary communication connections between GPU nodes for the training process as the connection relationships, a communication topology graph that perfectly matches the communication logic of the current training process is generated.

[0030] Specifically, the process execution information of the distributed training service is collected, as well as the necessary communication dependencies between nodes. The collected information includes, but is not limited to: parallel mode configuration information of training tasks, deployment information of training processes on each GPU node, communication connection establishment information between nodes, communication traffic statistics, timing information of communication interactions, and micro-batch scheduling information of training tasks.

[0031] For example, the above information can be collected through the management interface of the training framework, through mirror collection of traffic collection devices in the cluster network, or through local collection of agent programs deployed on each GPU node. This application embodiment does not limit the specific collection method, as long as the necessary communication dependencies between nodes of the training service can be fully obtained.

[0032] It should be noted that the business-essential communication dependencies, in this embodiment, specifically refer to the communication connections necessary for the normal operation of the training task. They do not include non-business-essential communication connections such as operation and maintenance management communication, heartbeat keep-alive communication, and hardware status acquisition communication between nodes. Only business-essential communication dependencies are collected, which can ensure that the generated communication topology map matches the actual operation logic of the training business and avoid non-business communication connections from interfering with subsequent topology robustness calculations.

[0033] For example, for the above-mentioned 100-node distributed training cluster, the collected training business process running information and communication dependencies are as follows: The training task adopts a hybrid parallel training mode. The business communication traffic between nodes is concentrated between two adjacent nodes in the same row and two adjacent nodes in the same column. The traffic of the four connections accounts for 100% in total, and there is no other business communication traffic.

[0034] In this embodiment, GPU nodes participating in the current training task are used as the basic units of the graph. Specifically, GPU nodes participating in the current training task are GPU nodes that have deployed the training process of the training task and participate in the training computation process. They do not include idle nodes, management nodes, storage nodes, log nodes, etc. in the cluster that are not participating in the training task. Using such GPU nodes as the basic units of the graph can ensure that the generated communication topology graph only covers nodes related to the training task and avoids irrelevant nodes from interfering with subsequent computations.

[0035] Specifically, each GPU node participating in the training process corresponds to a unique basic unit in the graph. Each basic unit has unique identification information, which corresponds one-to-one with the unique identifier of the corresponding GPU node. For example, the GPU node's IP address, hostname, device number within the cluster, row and column number, etc., can be used as the identification information of the basic unit. This application embodiment does not specifically limit this.

[0036] For example, for the cluster containing 100 GPU nodes participating in the training service, the generated communication topology diagram contains 100 corresponding basic units. Each basic unit corresponds one-to-one with a GPU node. The identification information of the basic unit adopts the row and column number of the GPU node, that is, (0,0) to (9,9), which is consistent with the GPU node number in the cluster.

[0037] It should be noted that training operations between GPU nodes require communication connections, which correspond to the connection relationship between the two basic units in the diagram. If there is a necessary communication dependency between two GPU nodes, a connection relationship is established between the two corresponding basic units.

[0038] Specifically, the connection can be directed or undirected, depending on the communication dependencies between nodes. For example, in pipelined parallel mode, the communication dependencies between nodes are unidirectional; the preceding computation node needs to transmit computation results to the subsequent computation node, while the subsequent node does not need to transmit business data to the preceding node. Therefore, the corresponding connection is a directed connection. In contrast, in data parallel mode, gradient synchronization between nodes is bidirectional, so the corresponding connection is an undirected connection. This application does not specifically limit the direction of the connection, as long as the business communication dependencies between nodes can be completely restored.

[0039] For example, for the aforementioned cluster of 100 nodes, based on the collected communication dependencies, a corresponding connection relationship is established for each basic unit. The specific rules are as follows: each basic unit establishes an undirected connection with two basic units in the same row whose column numbers are adjacent, with the column numbers cycling from one end to the other; that is, the left neighbor of column number 0 is column number 9, and the right neighbor of column number 9 is column number 0. Similarly, each basic unit establishes an undirected connection with two basic units in the same column whose row numbers are adjacent, with the row numbers cycling from one end to the other; that is, the upper neighbor of row number 0 is row number 9, and the lower neighbor of row number 9 is row number 0. According to this rule, each basic unit corresponds to 4 undirected connections, connecting 4 adjacent basic units, matching the collected inter-node communication traffic distribution. Taking the basic unit identified as (0,0) as an example, its four corresponding connections are connections to the four basic units (0,9), (0,1), (9,0), and (1,0), respectively. The weight of each connection is 0.25, which corresponds to the proportion of the communication traffic of the connection to the total service traffic of the node. The communication topology generated according to this rule is a two-dimensional toroidal topology structure with 10 rows and 10 columns. This application will not elaborate on this.

[0040] In this embodiment, the communication topology connectivity value is a crucial quantitative indicator for measuring the robustness of the entire training service communication topology. It determines the maximum number of GPU nodes that the training service can simultaneously handle while maintaining continuous service communication availability. It should be noted that the robustness of the entire communication topology is not determined by the node pair with the most communication paths, but rather by the node pair with the fewest communication paths. The overall topology's resilience to interruption is determined by the node pair with the weakest resilience.

[0041] In this embodiment of the application, the step of determining the communication topology connectivity value includes: Traverse all pairs of GPU nodes in the communication topology graph to generate a set of node pairs to be verified. For each node pair in the set of node pairs, determine the maximum number of independent communication paths between them that do not share any intermediate GPU nodes; The minimum value among the maximum number of independent communication paths corresponding to all node pairs is taken as the communication topology connectivity value.

[0042] Specifically, the process involves traversing all pairs of GPU nodes in the communication topology graph to generate a set of node pairs to be verified. In this embodiment, for the N basic units (corresponding to N GPU nodes) in the communication topology graph, the process involves traversing all combinations of two different basic units, with each combination corresponding to a set of GPU node pairs.

[0043] Specifically, if the connection relationship in the communication topology graph is a directed connection, then the node pair is an ordered node pair, that is, node pair (A, B) and node pair (B, A) are two different node pairs; if the connection relationship in the communication topology graph is an undirected connection, then the node pair is an unordered node pair, that is, node pair (A, B) and node pair (B, A) are the same node pair.

[0044] For example, in the communication topology graph of the above 100 nodes, the connection relationship is undirected. Therefore, the total number of unordered node pairs is 100 × 99 ÷ 2 = 4950. Traversing all 100 basic units, a set of 4950 node pairs to be verified is generated. The specific traversal process is as follows: starting from the basic unit of row number 0 and column number 0, it forms a node pair with the remaining 99 basic units from row number 0 and column number 1 to row number 9 and column number 9 in sequence; then, starting from the basic unit of row number 0 and column number 1, it forms a node pair with the remaining 98 basic units from row number 0 and column number 2 to row number 9 and column number 9 in sequence; and so on, until all combinations of two different basic units have been traversed. The final set of node pairs to be verified completely covers all the communication relationships between nodes in the communication topology graph without any omissions, ensuring that the communication topology connectivity value calculated later can truly reflect the minimum robustness level of the entire topology.

[0045] In this embodiment, for each node pair in the node pair set, determining the maximum number of independent communication paths between them that do not share any intermediate GPU nodes requires first clarifying the definitions of communication path, intermediate node, and independent communication path. For a node pair (source node, target node), the communication path between them, in this embodiment, is specifically a complete connection link starting from the basic unit corresponding to the source node, following the connection relationship in the diagram, and reaching the basic unit corresponding to the target node. This link consists of a series of continuous connection relationships. The basic units included in the link, excluding the source node and the target node, are the intermediate nodes of this communication path. The source node and the target node are not intermediate nodes. For example, for the node pair (0,0) and (0,2) in the above 100-node topology, there exists a connection link (0,0)→(0,1)→(0,2). This link is a communication path between node (0,0) and node (0,2), where node (0,1) is the intermediate node of this communication path.

[0046] It should be noted that the two communication paths are independent communication paths that do not share any intermediate GPU nodes. Specifically, in this embodiment, these two communication paths have no common basic units other than the source node and the target node, that is, they have no common intermediate GPU nodes. For example, for the node pair (0,0) and (0,2) in the above 100-node topology, the first communication path is (0,0)→(0,1)→(0,2), with the intermediate node being (0,1); the second communication path is (0,0)→(1,0)→(1,1)→(1,2)→(0,2), with the intermediate nodes being (1,0), (1,1), and (1,2). These two paths have no common intermediate nodes other than the source node (0,0) and the target node (0,2), therefore these two paths are independent communication paths. If the second communication path is (0,0)→(0,1)→(1,1)→(1,2)→(0,2), then this path shares the intermediate node (0,1) with the first path. Therefore, these two paths are not independent communication paths.

[0047] The core concept of this application is based on the following discovery: If two communication paths share an intermediate node, both communication paths will be interrupted simultaneously when the shared intermediate node goes offline, thus failing to provide redundancy backup. However, for independent communication paths, the offline status of the intermediate node in one path will not affect the communication availability of the other independent paths. In other words, the maximum number of independent communication paths between node pairs determines the resilience of the node pair; the more independent paths there are, the stronger the resilience.

[0048] Specifically, for a pair of nodes, determine the maximum number of independent communication paths between them that do not share any intermediate GPU nodes. The specific implementation is as follows: first, enumerate all possible communication paths between the pair of nodes; then, filter out the path combinations that do not share any intermediate nodes from all paths; traverse all possible path combinations and find the combination that contains the most paths. The number of paths contained in this combination is the maximum number of independent communication paths between the pair of nodes.

[0049] For example, in the above 100-node topology, for the weakest neighboring node pair (0,0) and (0,1) with the least resistance to interruption, after enumerating all possible communication paths and filtering, at most 4 independent communication paths can be found that do not share any intermediate nodes, as follows: The first path is (0,0)→(0,1), with no intermediate nodes; The second path is (0,0)→(1,0)→(2,0)→(3,0)→(4,0)→(5,0)→(6,0)→(7,0)→(8,0)→(9,0)→(9,1)→(8,1)→(7,1)→(6,1)→(5,1)→(4,1)→(3,1)→(2,1)→(1,1)→(0,1), with intermediate nodes being (1,0), (2,0), (3,0), (4,0), (5,0), (6,0), (7,0), (8,0), (9,0), (9,1), (8,1), (7,1), (6,1), (5,1), (4,1), (3,1), (2,1), (1,1); The third path is (0,0)→(9,0)→(9,9)→(9,8)→(9,7)→(9,6)→(9,5)→(9,4)→(9,3)→(9,2)→(9,1)→(8,1)→(7,1)→(6,1)→(5,1)→(4,1)→(3,1)→(2,1)→(1,1)→(0,1), with intermediate nodes being (9,0), (9,9), (9,8), (9,7), (9,6), (9,5), (9,4), (9,3), (9,2), (9,1), (8,1), (7,1), (6,1), (5,1), (4,1), (3,1), (2,1), (1,1); The fourth path is (0,0)→(1,0)→(2,0)→(3,0)→(4,0)→(5,0)→(6,0)→(7,0)→(8,0)→(8,1)→(7,1)→(6,1)→(5,1)→(4,1)→(3,1)→(2,1)→(1,1)→(0,1), with intermediate nodes (1,0), (2,0), (3,0), (4,0), (5,0), (6,0), (7,0), (8,0), (8,1), (7,1), (6,1), (5,1), (4,1), (3,1), (2,1), (1,1).

[0050] The above four paths, apart from the source node (0,0) and the target node (0,1), have no shared intermediate nodes, and are therefore independent communication paths. After traversing all possible path combinations, it is impossible to find a combination containing 5 or more independent paths. Therefore, the maximum number of independent communication paths between the node pair (0,0) and (0,1) is 4.

[0051] For example, for the node pair (0,0) and (5,5) in the above 100-node topology, after enumeration and filtering, a maximum of 6 independent communication paths can be found that do not share any intermediate nodes, and the number of independent paths is greater than 4; for the node pair (0,0) and (9,9), a maximum of 5 independent communication paths can be found, and the number of independent paths is also greater than 4. Traversing all 4950 node pairs in the set of node pairs to be verified, the maximum number of independent communication paths for all node pairs is ≥ 4. Among them, the maximum number of independent communication paths for all adjacent node pairs is 4, which is the minimum value among all node pairs. Of course, in other embodiments, for directed communication dependencies or other dependencies not in the above embodiments, the minimum value among all node pairs is not limited to 4. The above is just one example, and other examples are not elaborated here.

[0052] In this embodiment, the anti-interruption capability of the entire communication topology is determined by the node pair with the weakest anti-interruption capability. Once the communication of this node pair is interrupted, the entire training service will not be able to run normally. Therefore, using the minimum number of independent paths among all node pairs as the connectivity value of the topology can ensure that the fault tolerance threshold determined later can cover the communication security requirements of all node pairs and will not result in the communication interruption of some node pairs.

[0053] For example, in the communication topology graph of the 100 nodes mentioned above, the minimum number of independent communication paths among all node pairs is 4. Therefore, the communication topology connectivity value of this topology graph is 4. It should be noted that the physical meaning of the connectivity value is that the entire communication topology can have a maximum of 3 nodes removed at the same time, and the connectivity value of the remaining topology is still ≥1. All node pairs have at least one usable communication path, and the training service can run normally. If 4 or more nodes are removed at the same time, the connectivity value of the remaining topology may drop to 0, and there is a risk that all communication paths between some node pairs will be interrupted, causing the training service to fail to run normally.

[0054] In this embodiment, the connectivity fault tolerance threshold is a key constraint in the entire repair process. It limits the maximum number of GPU nodes that can be repaired simultaneously in a single batch, ensuring that the communication topology connectivity value of the training service is always higher than the minimum requirement for normal service operation during the repair process, thus avoiding service communication interruption caused by the repair action.

[0055] In this embodiment of the application, determining the connectivity fault tolerance threshold based on the communication topology graph includes: Obtain the set values ​​of the communication topology connectivity required for the normal operation of the current training service; The maximum number of GPU nodes that can temporarily detach from the communication topology map in a single batch is calculated when the communication topology connectivity value is continuously higher than a set value. This maximum number is then determined as the connectivity fault tolerance threshold for the current training service.

[0056] Specifically, the set value of the communication topology connectivity required for the normal operation of the current training service is, in this embodiment of the application, the threshold corresponding to the minimum communication topology connectivity required for the training service to maintain continuous normal operation. This set value is determined by the communication requirements of the training service.

[0057] It's important to note that the core prerequisite for the normal operation of the training service is that there must always be at least one available communication path between any two nodes that have communication dependencies. This means the minimum communication topology connectivity value required for the training service is 1. Therefore, setting the value to 0 and maintaining a communication topology connectivity value higher than the set value is equivalent to having a communication topology connectivity value ≥ 1, thus meeting the normal operation requirements of the training service. For example, for the aforementioned training service with 100 nodes, the minimum connectivity value required for its normal operation is 1, therefore the set value for the obtained communication topology connectivity value is 0.

[0058] In this embodiment of the application, the maximum number of GPU nodes that can be temporarily removed from the communication topology graph in a single batch is calculated when the communication topology connectivity value is kept higher than the set value. Specifically, the connectivity value of the original communication topology is C, and the set value is S. To keep the remaining topology connectivity value after removing nodes always higher than S, that is, the remaining topology connectivity value is ≥ S+1, the maximum number of nodes that can be temporarily removed in a single batch is C-(S+1).

[0059] For example, in the aforementioned communication topology of 100 nodes, the original connectivity value C is 4, and the set value S is 0. Therefore, the maximum number of nodes that can be temporarily removed in a single batch is 4 - (0 + 1) = 3. In this way, if 3 nodes are temporarily removed from the topology at the same time, the connectivity value of the sub-topology consisting of the remaining 97 nodes is still ≥1, which is higher than the set value of 0, meeting the normal operation requirements of the training service. If 4 or more nodes are removed at the same time, the connectivity value of the remaining topology may drop to 0, which will not meet the service operation requirements.

[0060] Specifically, the maximum number of GPU nodes that can be temporarily detached in a single batch, calculated above, is determined as the connectivity fault tolerance threshold for the current training service. For example, for the aforementioned training service with 100 nodes, the connectivity fault tolerance threshold is 3, meaning that a maximum of 3 GPU nodes can be repaired simultaneously in the same batch. It should be noted that this threshold is calculated based on the robustness of the actual communication topology of the current training service, matching the communication requirements of the service and avoiding the problem of the number of nodes repaired in a single batch exceeding the service's capacity.

[0061] In this embodiment of the application, all repairable faulty GPU nodes are divided into batches to generate a set of faulty nodes corresponding to each repair batch, including: Based on the error information reported by the repairable faulty GPU nodes, determine the fault risk level corresponding to each faulty node; Sort all repairable faulty GPU nodes in descending order of fault risk level; With the constraint that the number of faulty nodes included in a single batch does not exceed the connectivity fault tolerance threshold of the current training service, faulty nodes are sequentially assigned to the corresponding repair batches according to the sorting results, generating a set of faulty nodes corresponding to each repair batch.

[0062] Specifically, based on the error information reported by the repairable faulty GPU nodes, the fault risk level of each faulty node is determined. The error information reported by the repairable faulty GPU nodes includes, but is not limited to, fault error codes, hardware operating status data, cumulative number of errors, and fault duration. The fault risk level is determined by the probability of the fault escalating to an unrepairable fault. The more cumulative the errors and the longer the fault duration, the higher the probability of fault escalation, the higher the corresponding fault risk level, and the higher the repair priority.

[0063] For example, regarding the 20 faulty GPU nodes in the aforementioned 100-node cluster, the error information collected through the GPU management interface shows that the cumulative number of correctable memory errors of the 20 nodes, sorted from high to low, is consistent with the order of node numbers (0,0), (0,5), (1,3), (2,7), (3,2), (3,8), (4,1), (4,6), (5,4), (5,9), (6,0), (6,5), (7,3), (7,7), (8,2), (8,8), (9,1), (9,4), (9,6), (9,9). Therefore, the fault risk level of the 20 faulty nodes is determined according to this order, with higher risk levels and higher repair priorities for nodes ranked earlier.

[0064] In this embodiment, the number of faulty nodes included in a single batch does not exceed the connectivity fault tolerance threshold of the current training service. Faulty nodes are sequentially assigned to corresponding repair batches according to their sorting results, generating a set of faulty nodes for each repair batch. For example, for the 20 faulty nodes in the aforementioned 100-node cluster, the connectivity fault tolerance threshold is 3, meaning the maximum number of faulty nodes included in a single batch is 3. The nodes are sorted from highest to lowest fault risk level, such as... Figures 2 to 4 ,in Figure 2 This is a schematic diagram of the first batch of repairs. Figure 3 This is a schematic diagram of the second batch of repairs. Figure 4This is a schematic diagram of the first repair batch. The black nodes represent faulty nodes included in the current repair batch. The specific batch division process is as follows: The first batch of repairs included the top 3 faulty nodes in the sorted order, which were (0,0), (0,5), and (1,3). The number of nodes included was 3, which did not exceed the connectivity fault tolerance threshold of 3 and met the constraint requirements. The second repair batch included the faulty nodes ranked 4th to 6th, which were (2, 7), (3, 2), and (3, 8) respectively. The number of nodes included was 3, which meets the constraint requirements. The third repair batch included the faulty nodes ranked 7th to 9th, which were (4,1), (4,6), and (5,4) respectively. The number of nodes included was 3, which meets the constraint requirements. The fourth repair batch included the faulty nodes ranked 10th to 12th, which were (5, 9), (6, 0), and (6, 5) respectively. The number of nodes included was 3, which met the constraint requirements. The fifth repair batch included the faulty nodes ranked 13th to 15th, which are (7,3), (7,7), and (8,2) respectively. The number of nodes included is 3, which meets the constraint requirements. The sixth repair batch included the faulty nodes ranked 16th to 18th, which are (8,8), (9,1), and (9,4) respectively. The number of nodes included is 3, which meets the constraint requirements. The seventh repair batch included the remaining faulty nodes ranked 19th to 20th, namely (9, 6) and (9, 9), with a total of 2 nodes included. This does not exceed the connectivity fault tolerance threshold of 3, and meets the constraint requirements.

[0065] It should be noted that the fourth to seventh repair batches were carried out in accordance with... Figures 2 to 4 The principle of iteration is not illustrated in this application. Following the above process, a set of faulty nodes corresponding to 7 repair batches is finally generated, completing the batch division of all repairable faulty GPU nodes.

[0066] It should be noted that the batch division process uses the connectivity fault tolerance threshold as a key constraint to ensure that the number of faulty nodes included in each batch does not exceed the safe range that the training service can tolerate, thus avoiding the risk of service communication interruption caused by repair actions. At the same time, prioritizing the repair of nodes with higher risk levels according to fault risk level can minimize the probability of fault escalation and improve the stability of cluster operation.

[0067] In this embodiment of the application, before performing fault repair according to the generated repair batch, the following steps are included: Lock the computation execution timing and communication interaction rules for the current training service; For the set of faulty nodes currently awaiting repair, verify that after removing the corresponding node, the connectivity of the communication topology formed by the remaining healthy GPU nodes is still higher than the set value, confirming that all necessary communication paths between the remaining healthy nodes remain available.

[0068] Specifically, the computation execution timing and communication interaction rules of the current training service are locked. In this embodiment, before the node repair of the current batch is performed, the main operating parameters such as the micro-batch scheduling timing, gradient synchronization cycle, pipeline stage connection timing, and triggering timing of inter-node communication interaction of the training task are fixed to ensure that the communication logic of the training service will not change during the node repair process, and to avoid problems such as timing disorder and communication loss caused by temporary node offline.

[0069] For example, for the training service with the above 100 nodes, before the first batch of repairs is performed, the single-round iteration cycle, gradient synchronization cycle, and pipeline micro-batch scheduling interval of the training task are locked, and the triggering time of inter-node communication is bound to the number of iteration steps. During the repair process, the above parameters will not change.

[0070] For example, for the first repair batch of the aforementioned 100-node cluster, the corresponding set of faulty nodes is (0,0), (0,5), and (1,3). Before performing the repair, the three nodes are temporarily removed from the communication topology graph. The communication topology connectivity value of the sub-topology composed of the remaining 97 healthy nodes is calculated. The calculation result is 2, which is higher than the set value of 0, thus meeting the operational requirements of the training service. Then, all node pairs in the remaining sub-topology are traversed to verify that there is at least one available communication path between all node pairs and that all necessary communication paths for the service remain available. After verification, the repair operation for the current batch is started.

[0071] It should be noted that this pre-verification step can perform secondary verification on each batch of repair actions, ensuring that even if there are temporary changes in the topology, the repair actions will not cause business communication interruptions, thus further improving the reliability of the solution.

[0072] In this embodiment of the application, fault repair is performed on the corresponding set of faulty nodes sequentially according to the generated repair batches, including: Get the set of fault nodes corresponding to the current repair batch, and the fault type corresponding to each fault node; Based on the fault type, a hierarchical repair operation sequence with invasiveness ranging from low to high is matched, and the corresponding repair operation is executed to clear the node fault. Confirm that the repair operations for all faulty nodes in the current batch have been completed, and output the batch repair results.

[0073] Specifically, the hierarchical repair operation sequence is a set of repair operations ordered from low to high according to their invasiveness to the node's running state. The lower the invasiveness, the less impact it has on the cluster operation. Prioritizing the execution of repair operations with low invasiveness can minimize the impact of repair actions on training services.

[0074] For example, for the faulty node in the aforementioned 100-node cluster whose fault type is "accumulated correctable memory errors exceeding the limit," the matching hierarchical repair operation sequence, from low to high, is as follows: memory fault page isolation operation, GPU firmware status refresh operation, GPU driver parameter reset operation, GPU soft reboot operation, and node operating system soft reboot operation. When performing repair operations, they are executed sequentially from low to high until the node fault is cleared, without needing to perform more invasive operations, thus minimizing the impact of repair on the node.

[0075] For example, for the three faulty nodes in the first repair batch of the aforementioned 100-node cluster, all of them had accumulated correctable errors in the video memory. Therefore, a corresponding hierarchical repair operation sequence was matched. First, the video memory error page isolation operation was performed in parallel on the three nodes to isolate the video memory physical pages with errors. Then, the GPU firmware status refresh operation was performed to clear the accumulated error codes. After completion, the fault status of the node was verified to confirm that the fault had been cleared. No more invasive operations were required. The execution time of the entire repair operation was fixed. The three nodes were executed in parallel, and the total time was consistent with the repair time of a single node.

[0076] In this embodiment of the application, after completing the repair operation of a single batch of faulty nodes, the process includes: Perform hardware status verification to collect data and confirm that the fault of the GPU node has been cleared after repair, and / or the hardware performance has been restored to the rated standard; Perform a topology connectivity check to confirm that after the repair node reconnects, the communication topology connectivity value of the current training service is higher than the set value; Perform a business operation verification to confirm that the computation and communication processes of the training business are normal after the repair node is connected. After all verifications pass, restore the training business participation permissions of the corresponding GPU node.

[0077] Specifically, hardware status verification, in this embodiment, involves a comprehensive verification of the hardware operating status of the repaired GPU node. Verification includes, but is not limited to, whether fault error codes have been cleared, whether memory read / write bandwidth has returned to its rated standard, whether floating-point computing power has returned to its rated standard, and whether hardware temperature and power consumption are within the normal range. Only when all verification items meet the requirements is the hardware status verification considered successful. For example, regarding node (0,0) in the first batch of repairs, before repair, the cumulative number of correctable errors in the memory exceeded the standard; after repair, the error codes were cleared, memory read / write bandwidth and floating-point computing power both returned to their rated standards, and hardware temperature and power consumption were within the normal range, thus the hardware status verification passed.

[0078] For example, in this embodiment, topology connectivity verification specifically involves recalculating the connectivity value of the entire communication topology after the repaired node is reconnected to the communication topology graph. The connectivity value is confirmed to be higher than the set value of 0. Simultaneously, the communication connection between the repaired node and its adjacent nodes is verified to be normal, and the packet loss rate is lower than a preset threshold, ensuring that the robustness of the entire topology returns to normal after the node is reconnected. For the three nodes in the first batch of repairs, after reconnecting to the topology, the calculated communication topology connectivity value of the entire 100-node topology is 4, higher than the set value of 0. The communication connection between the three nodes and their respective adjacent nodes is normal, and the packet loss rate is lower than the preset threshold; therefore, the topology connectivity verification passes.

[0079] Specifically, in this embodiment, the business operation verification involves reconnecting the repaired nodes to the training task and verifying that the nodes can participate normally in the training computation process. This verifies that business communication processes such as gradient synchronization, intermediate result transmission, and pipeline data interaction between nodes are normal, and that key operational indicators such as the iteration cycle and convergence speed of the training task are normal, ensuring that the node connection will not negatively impact the training business. For example, after the three nodes in the first batch of repairs regained their training business participation permissions, the single-round iteration cycle of the training task remained stable, the gradient synchronization process was normal, and there was no gradient jitter; therefore, the business operation verification passed.

[0080] In this embodiment of the application, obtaining error information reported by a repairable faulty GPU node includes: The fault error codes and hardware operating status data reported by each GPU node are collected in real time through the GPU management interface. Based on a pre-defined library of repairable fault types, the collected error information is assessed for repairability, and GPU nodes that meet the repairability criteria are selected. The library of repairable fault types includes fault types and error codes corresponding to correctable memory errors, driver malfunctions, and firmware status anomalies.

[0081] It should be noted that the GPU management interface is a hardware status management interface provided by the GPU chip manufacturer. It can collect real-time hardware operation data such as GPU memory error information, temperature, power consumption, computing power utilization, driver running status, and firmware status. The collection frequency can be configured according to needs to ensure timely detection of node fault status. The recoverable fault type library is built based on fault data accumulated during long-term cluster operation and maintenance. The library contains all fault types that can be cleared through operation and maintenance operations without replacing hardware, as well as corresponding fault error codes, judgment rules, and repair operation sequences. It can realize automatic determination of fault recoverability without manual intervention, improving the automation level of fault handling.

[0082] For example, for the aforementioned 100-node cluster, the operation and maintenance system collects fault error codes and hardware operating status data of the 100 GPU nodes in real time through the GPU management interface. If the error codes of 20 nodes are found to correspond to an excessive number of correctable errors in the video memory, the system queries the repairable fault type library. If the fault type is found to be a repairable fault, these 20 nodes are identified as repairable fault GPU nodes and included in the batch classification. If the error codes of a node are found to correspond to physical damage to the video memory hardware, and this fault type is not in the repairable fault type library, it is identified as an unrepairable fault and is not included in the batch classification, triggering a hardware replacement alarm.

[0083] In addition, the embodiments of this application also include corresponding topology and threshold dynamic adjustment steps, specifically, real-time monitoring of the life cycle changes, parallel mode adjustments, node status changes and communication topology changes of the distributed training services in the cluster, synchronously updating the communication topology graph and communication topology connectivity value of the corresponding training services, recalculating the connectivity fault tolerance threshold, dynamically adjusting the division rules of subsequent repair batches, and adapting to the real-time changes in the cluster status.

[0084] Those skilled in the art will understand that during the long-term training of a large model, the parallel mode of the training task, the number of nodes participating in the training, and the communication topology may all be adjusted, and the corresponding topology connectivity value and fault tolerance threshold will also change. By dynamically adjusting the steps, it is possible to ensure that the repair scheme always adapts to the real-time status of the training service and guarantees the security of the repair process.

[0085] For example, in the training task of the above 100 nodes, the configuration of the hybrid parallel mode was adjusted during the training process. The corresponding communication topology was changed from a 10×10 two-dimensional torus to a ring topology with 20 nodes per group. The connectivity value of each group of topologies is 2, and the corresponding set value is still 0. Therefore, the recalculated connectivity fault tolerance threshold is 1. The subsequent fault batch division rule was adjusted to include a maximum of 1 fault node per batch.

[0086] This application further provides a fault repair system for a GPU graphics processor, such as... Figure 5 As shown, the fault repair system is applied to a distributed training cluster containing multiple GPU nodes, where each GPU node communicates with at least two GPU nodes to form a communication topology. The determination module determines a connectivity fault tolerance threshold based on the communication topology graph. The connectivity fault tolerance threshold is the maximum number of GPU nodes that can be repaired simultaneously in the same batch while keeping the current training service communication topology connectivity value higher than a set value. The communication topology connectivity value is the minimum value among the maximum number of independent communication paths corresponding to all node pairs in the communication topology graph, and the set value is greater than or equal to zero. The batch partitioning module divides all repairable faulty GPU nodes into batches based on the error information reported by the repairable faulty GPU nodes and the connectivity fault tolerance threshold, generating a set of faulty nodes corresponding to each repair batch. The repair module repairs the corresponding set of faulty nodes sequentially according to the generated repair batches.

[0087] Since the principle behind this system's problem-solving is similar to the methods described above, the implementation of this system can be found in the implementation of the methods, and will not be repeated here.

[0088] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for troubleshooting GPU graphics processors, applied to a distributed training cluster containing multiple GPU nodes, wherein each GPU node communicates with at least two GPU nodes to form a communication topology, characterized in that, The fault repair method includes: Based on the communication topology graph, a connectivity fault tolerance threshold is determined. The connectivity fault tolerance threshold is the maximum number of GPU nodes that can be repaired simultaneously in the same batch while keeping the current training service communication topology connectivity value higher than a set value. The communication topology connectivity value is the minimum value among the maximum number of independent communication paths corresponding to all node pairs in the communication topology graph. The set value is greater than or equal to zero. Based on the error information reported by the repairable faulty GPU nodes and the connectivity fault tolerance threshold, all repairable faulty GPU nodes are divided into batches to generate a set of faulty nodes corresponding to each repair batch. The corresponding set of faulty nodes will be repaired sequentially according to the generated repair batches.

2. The GPU graphics processor fault repair method according to claim 1, characterized in that, The steps for establishing the communication topology map include: Collect process execution information of distributed training services, as well as the necessary communication dependencies between nodes; Using the GPU nodes participating in the current training process as the basic units of the graph, and the necessary communication connections between GPU nodes for the training process as the connection relationships, a communication topology graph that perfectly matches the communication logic of the current training process is generated.

3. The GPU graphics processor fault repair method according to claim 2, characterized in that, The steps for determining the communication topology connectivity value include: Traverse all pairs of GPU nodes in the communication topology graph to generate a set of node pairs to be verified. For each node pair in the set of node pairs, determine the maximum number of independent communication paths between them that do not share any intermediate GPU nodes; The minimum value among the maximum number of independent communication paths corresponding to all node pairs is taken as the communication topology connectivity value.

4. The GPU graphics processor fault repair method according to claim 3, characterized in that, Determining the connectivity tolerance threshold based on the communication topology graph includes: Obtain the set values ​​of the communication topology connectivity required for the normal operation of the current training service; The maximum number of GPU nodes that can temporarily detach from the communication topology map in a single batch is calculated when the communication topology connectivity value is continuously higher than a set value. This maximum number is then determined as the connectivity fault tolerance threshold for the current training service.

5. The GPU graphics processor fault repair method according to claim 1, characterized in that, All repairable faulty GPU nodes are divided into batches, generating a set of faulty nodes corresponding to each repair batch, including: Based on the error information reported by the repairable faulty GPU nodes, determine the fault risk level corresponding to each faulty node; Sort all repairable faulty GPU nodes in descending order of fault risk level; With the constraint that the number of faulty nodes included in a single batch does not exceed the connectivity fault tolerance threshold of the current training service, faulty nodes are sequentially assigned to the corresponding repair batches according to the sorting results, generating a set of faulty nodes corresponding to each repair batch.

6. The GPU graphics processor fault repair method according to claim 1, characterized in that, Before performing fault repair according to the generated repair batch, the following should be included: Lock the computation execution timing and communication interaction rules for the current training service; For the set of faulty nodes currently awaiting repair, verify that after removing the corresponding node, the connectivity of the communication topology formed by the remaining healthy GPU nodes is still higher than the set value, confirming that all necessary communication paths between the remaining healthy nodes remain available.

7. The GPU graphics processor fault repair method according to claim 1, characterized in that, After completing the repair operation for a single batch of faulty nodes, the following steps are included: Perform hardware status verification to collect data and confirm that the fault of the GPU node has been cleared after repair, and / or the hardware performance has been restored to the rated standard; Perform a topology connectivity check to confirm that after the repair node reconnects, the communication topology connectivity value of the current training service is higher than the set value; Perform a business operation verification to confirm that the computation and communication processes of the training business are normal after the repair node is connected. After all verifications pass, restore the training business participation permissions of the corresponding GPU node.

8. The GPU graphics processor fault repair method according to claim 1, characterized in that, Obtain error information reported by repairable faulty GPU nodes, including: The fault error codes and hardware operating status data reported by each GPU node are collected in real time through the GPU management interface. Based on a pre-defined library of repairable fault types, the collected error information is assessed for repairability, and GPU nodes that meet the repairability criteria are selected. The library of repairable fault types includes fault types and error codes corresponding to correctable memory errors, driver malfunctions, and firmware status anomalies.

9. The GPU graphics processor fault repair method according to claim 1, characterized in that, The faulty node sets are repaired sequentially according to the generated repair batches, including: Get the set of fault nodes corresponding to the current repair batch, and the fault type corresponding to each fault node; Based on the fault type, a hierarchical repair operation sequence with invasiveness ranging from low to high is matched, and the corresponding repair operation is executed to clear the node fault. Confirm that the repair operations for all faulty nodes in the current batch have been completed, and output the batch repair results.

10. A GPU graphics processor fault repair system, applied to a distributed training cluster containing multiple GPU nodes, wherein each GPU node communicates with at least two GPU nodes to form a communication topology, characterized in that, The fault repair system includes: The determination module determines a connectivity fault tolerance threshold based on the communication topology graph. The connectivity fault tolerance threshold is the maximum number of GPU nodes that can be repaired simultaneously in the same batch while keeping the current training service communication topology connectivity value higher than a set value. The communication topology connectivity value is the minimum value among the maximum number of independent communication paths corresponding to all node pairs in the communication topology graph, and the set value is greater than or equal to zero. The batch partitioning module divides all repairable faulty GPU nodes into batches based on the error information reported by the repairable faulty GPU nodes and the connectivity fault tolerance threshold, generating a set of faulty nodes corresponding to each repair batch. The repair module repairs the corresponding set of faulty nodes sequentially according to the generated repair batches.

Images