Method, device and electronic device for determining faulty graphics processing chips in a computing cluster

Abstract

The present disclosure provides a method, device and electronic device for determining a faulty graphics processing chip in a computing power cluster, and relates to technical fields such as generative large language models, graphics processing chips, computing power clusters, and collective communication operations. The method comprises: respectively obtaining the number of executed collective communication operations, channel numbers and execution status information of the currently executed collective communication operations corresponding to each GPU constituting the computing power cluster; for each GPU with the same channel number, determining a first diagnostic result based on the corresponding number of executed collective communication operations and execution status information; summarizing the first diagnostic results corresponding to each channel belonging to the same communication group to determine a second diagnostic result; summarizing the second diagnostic results corresponding to each communication group belonging to the same training task to determine a target diagnostic result; and determining the faulty GPU that affects the normal execution of the current training task based on the target diagnostic result. This method can accurately locate the faulty GPU that causes the training task to hang.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, specifically to the field of artificial intelligence technology such as generative large language models, graphics processing chips, computing power clusters, and collective communication operations, and especially to a method, device, electronic device, computer-readable storage medium, and computer program product for determining a faulty graphics processing chip in a computing power cluster. Background Art

[0002] In large-model training, due to the significant increase in model size and data volume, multiple compute nodes (a single GPU typically functions as a compute node) must collaborate to complete the training task. To achieve efficient data sharing and updates between nodes, compute nodes typically use the Collective Communication Library (CCL) for collective communication operations such as broadcast and all-reduce. This ensures that all participating nodes can complete parameter updates at each training step, thereby maintaining model consistency.

[0003] However, the synchronization mechanism of collective communication also requires coordination among all nodes during training. Any single node failure during large model training can impact the overall progress of the task. Specifically, if a single node becomes blocked or "hung" (which can be understood as suspended or stuck, preventing further execution), all other nodes will also be forced to "hang" while waiting for synchronization data, preventing the entire training task from continuing. However, such issues lack explicit signatures, making it difficult to quickly locate the faulty node. This is especially true in multi-node clusters, where the point of a hung node is difficult to identify, further complicating troubleshooting and severely impacting training efficiency and system stability. Summary of the Invention

[0004] The embodiments of the present disclosure provide a method, device, electronic device, computer-readable storage medium, and computer program product for determining a faulty graphics processing chip in a computing power cluster.

[0005] In the first aspect, an embodiment of the present disclosure proposes a method for determining a faulty graphics processing chip in a computing power cluster, including: respectively obtaining the number of executed collective communication operations, channel number and execution status information of the currently executed collective communication operation corresponding to each graphics processing chip GPU constituting the computing power cluster; for each GPU with the same channel number, determining a first diagnostic result corresponding to the channel level based on the corresponding number of executed collective communication operations and execution status information; summarizing the first diagnostic results corresponding to each channel belonging to the same communication group, and determining a second diagnostic result corresponding to the communication group level; summarizing the second diagnostic results corresponding to each communication group belonging to the same training task, and determining a target diagnostic result corresponding to the training task level; and determining the faulty GPU that affects the normal execution of the current training task based on the target diagnostic result.

[0006] In the second aspect, an embodiment of the present disclosure proposes a device for determining a faulty graphics processing chip in a computing power cluster, comprising: an information acquisition unit, configured to respectively acquire the number of executed collective communication operations, channel number and execution status information of the currently executed collective communication operation corresponding to each graphics processing chip GPU constituting the computing power cluster; a channel-level diagnostic result determination unit, configured to determine, for each GPU having the same channel number, a first diagnostic result corresponding to the channel level based on the corresponding number of executed collective communication operations and execution status information; a communication group-level diagnostic result determination unit, configured to summarize the first diagnostic results corresponding to each channel belonging to the same communication group, and determine the second diagnostic result corresponding to the communication group level; a training task-level diagnostic result determination unit, configured to summarize the second diagnostic results corresponding to each communication group belonging to the same training task, and determine the target diagnostic result corresponding to the training task level; a faulty GPU locating unit, configured to determine the faulty GPU that affects the normal execution of the current training task based on the target diagnostic result.

[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that when the at least one processor executes, it can implement the method for determining a faulty graphics processing chip in a computing power cluster as described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the method for determining a faulty graphics processing chip in a computing power cluster as described in the first aspect when executed.

[0009] In a fifth aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the steps of the method for determining a faulty graphics processing chip in a computing power cluster as described in the first aspect.

[0010] The solution for determining the faulty graphics processing chip in a computing cluster provided by the present disclosure obtains the number of collective communication operations executed, the channel number to which it belongs, and the execution status information of the currently executed collective communication operation for each GPU in a computing cluster composed of multiple GPUs, and performs a multi-level and multi-dimensional comprehensive analysis of the information related to the executed collective communication operations of each GPU. Finally, the faulty GPU that causes the current training task to hang can be determined, thereby improving the efficiency of troubleshooting faulty nodes in multi-node computing clusters, minimizing the duration of training tasks hanging due to faults, and ultimately reducing the overall time required to execute a single training task.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Other features, objects and advantages of the present disclosure will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following drawings:

[0013] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0014] Figure 2 A flowchart of a method for determining a faulty graphics processing chip in a computing cluster provided by an embodiment of the present disclosure;

[0015] Figure 3 A flowchart of a method for diagnosing a faulty GPU at the channel level provided in an embodiment of the present disclosure;

[0016] Figure 4-1 Two branch diagrams for obtaining the first diagnosis results under different circumstances provided by the embodiment of the present disclosure;

[0017] Figure 4-2 A schematic diagram of counter features of each GPU corresponding to the symmetric combined communication operation provided in an embodiment of the present disclosure;

[0018] Figure 4-3 A schematic diagram of counter features of each GPU corresponding to an asymmetric combined communication operation provided in an embodiment of the present disclosure;

[0019] Figure 5 A flowchart of a method for summarizing and eliminating erroneous diagnosis results by obtaining intersections provided in an embodiment of the present disclosure;

[0020] Figure 6 A flowchart of a fault repair method for a faulty GPU provided by an embodiment of the present disclosure;

[0021] Figure 7-1 A structural flow chart of another system for determining a faulty graphics processing chip in a computing cluster provided by an embodiment of the present disclosure;

[0022] Figure 7-2 A schematic diagram of the parallel structure between GPUs in a computing cluster provided by an embodiment of the present disclosure;

[0023] Figure 8 This is a structural block diagram of a device for determining a faulty graphics processing chip in a computing cluster provided by an embodiment of the present disclosure;

[0024] Figure 9 A schematic diagram of the structure of an electronic device suitable for executing a method for determining a faulty graphics processing chip in a computing power cluster, provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other unless there is a conflict.

[0026] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0027] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the method, apparatus, electronic device, and computer-readable storage medium for determining a faulty graphics processing chip in a computing cluster disclosed herein may be applied.

[0028] like Figure 1 As shown, the system architecture 100 may include a server 110, a network 120, and a computing cluster 130. The computing cluster 130 includes multiple graphics processors, such as 131, 132, 133, 134, and 135 (not just 5, Figure 1Only a few are used as examples), and the graphics cards 131, 132, 133, 134 and 135 include at least one graphics processing chip (i.e., GPU), that is, the computing power cluster 130 is a computing power cluster with multiple computing nodes; the network 120 is used as a medium for providing a communication link between the server 110 and the computing power cluster 130, and the network 120 may include various connection types, such as wired, wireless communication links or fiber optic cables, etc.

[0029] Users can use terminal devices to interact with server 110 via network 120 to receive or send messages, etc. Server 110, computing cluster 130, and terminal devices can all be installed with various applications for enabling information communication between them, such as applications for troubleshooting GPU failures, issuing training tasks, and instant messaging applications.

[0030] The terminal device and server 110 can be either hardware or software. When the terminal device is hardware, it can be any electronic device with a display screen, including but not limited to smartphones, tablets, laptops, and desktop computers. When the terminal device is software, it can be installed in any of the aforementioned electronic devices. It can be implemented as multiple software programs or software modules, or as a single software program or software module, without specific limitations here. When the server 110 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server 110 is software, it can be implemented as multiple software programs or software modules, or as a single software program or software module, without specific limitations here. The graphics processor that constitutes the computing power cluster 130 is typically hardware, but in special scenarios (such as simulation scenarios), it can also be software or a product of software execution, without specific limitations here.

[0031] The server 110 can provide various services through various built-in applications. Taking the faulty GPU troubleshooting application that can provide faulty GPU troubleshooting service as an example, the server 110 can achieve the following effects when running the faulty GPU troubleshooting application: first, the number of executed collective communication operations, channel number and execution status information of the currently executed collective communication operation corresponding to each GPU constituting the computing power cluster 130 are obtained through the network 120; then, for each GPU with the same channel number, based on the corresponding number of executed collective communication operations and execution status information, the first diagnostic result corresponding to the channel level is determined; then, the first diagnostic results corresponding to each channel belonging to the same communication group are summarized, and the second diagnostic result corresponding to the communication group level is determined; next, the second diagnostic results corresponding to each communication group belonging to the same training task are summarized, and the target diagnostic result corresponding to the training task level is determined; finally, the faulty GPU that affects the normal execution of the current training task is determined based on the target diagnostic result.

[0032] Furthermore, after determining the faulty GPU, the server 110 can also send the information to the computing cluster 130 through the network 120, so that the computing cluster 130 can try to use a feasible troubleshooting solution to repair the faulty GPU and resume the training task as soon as possible.

[0033] The methods for determining faulty graphics processing chips within a computing cluster, as provided in the subsequent embodiments of this disclosure, are generally performed by a server 110 with relatively strong computing power and abundant computing resources. Accordingly, the training strategy determination device for a computing cluster that uses a mix of chip models is also generally located within server 110. Of course, the method can also be performed by a device with similar computing power within or near computing cluster 130.

[0034] It should be understood that Figure 1 The number of servers, networks, computing clusters, and graphics cards shown in the figure is merely illustrative. Any number of servers, networks, computing clusters, and graphics cards may be used as needed.

[0035] Please refer to Figure 2 , Figure 2 This is a flow chart of a method for determining a faulty graphics processing chip in a computing cluster provided by an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0036] Step 201: respectively obtain the number of executed collective communication operations, the channel number, and the execution status information of the currently executed collective communication operation corresponding to each GPU constituting the computing power cluster;

[0037] This step is intended to determine the execution subject of the method by the faulty graphics processing chip in the computing power cluster (for example Figure 1 The server 110 shown in the figure obtains the number of executed collective communication operations, the channel number and the execution status information of the currently executed collective communication operation corresponding to each GPU constituting the computing power cluster.

[0038] Among them, the number of executed collective communication operations refers to the total number of communication operations that have been completed by the GPU during the execution process. These communication operations may include tasks such as data transmission, synchronization, and collaborative computing. The workload and performance of each GPU may be affected by the number of these operations. The number of operations executed can help evaluate the task progress and current load status of each GPU. In multi-GPU parallel computing systems, communication operations are usually an important part of the computing process. Especially in large-scale parallel computing, efficient communication between GPUs is key to ensuring system performance. By tracking the number of communication operations executed by each GPU, computing resources can be better managed, and computing task scheduling and load balancing can be optimized.

[0039] The channel number corresponds to the identifier of the physical or virtual communication channel used by each GPU when communicating. Typically, in distributed computing or multi-GPU systems, communication channels are used for data transmission paths or the transmission of synchronization signals. Different channels can be used for different data streams or operation types. In a multi-GPU parallel computing system, different GPUs may need to exchange data through multiple physical or logical channels. The channel number is a unique identifier that distinguishes these channels and can help the system accurately specify and monitor different communication paths during execution. Channel management is crucial to improving data transmission efficiency, avoiding bottlenecks, and reducing conflicts.

[0040] The execution status information of the currently executed collective communication operation may include three states:

[0041] 1) Not started: indicates that the current communication operation has not been started;

[0042] 2) Executing: Indicates that the current communication operation is in progress, which may be the process of data transmission or computing collaboration;

[0043] 3) Execution completed: indicates that the communication operation has been completed and the related data or synchronization operation has been successfully completed.

[0044] The execution status of collective communication operations is crucial for dynamic scheduling and resource management. In multi-GPU parallel computing systems, monitoring the execution status of collective communication operations helps the system determine which GPUs have completed their tasks, which are awaiting communication results, and which have not yet begun processing. Based on this information, the upper-level scheduling system can make intelligent decisions to avoid resource waste and performance bottlenecks, thereby improving overall system efficiency.

[0045] Step 202: For each GPU having the same channel number, determine a first diagnostic result corresponding to the channel level based on the corresponding executed collective communication operations and execution state information;

[0046] Based on step 201 , this step aims to determine, by the execution subject, a first diagnostic result corresponding to the channel level for each GPU having the same channel number based on the corresponding executed collective communication operations and execution status information.

[0047] For training tasks assigned to a computing cluster, they are typically divided into multiple communication groups, each of which contains multiple channels, and each GPU may be in different channels simultaneously. Therefore, the channel-level faulty GPU diagnosis performed in this step is for each GPU with the same channel number. This involves obtaining the number of collective communication operations executed by each GPU and the execution status of the currently executing collective communication operation, and then comparing this information to ultimately determine the first diagnostic result.

[0048] Generally speaking, for each GPU in any channel, the GPU with the smallest number of executed aggregate communication operations is most likely to be the faulty GPU, and the probability of other GPUs closely associated with it being the faulty GPU is also relatively high. Similarly, the GPU that does not have the same execution state information as other GPUs is also most likely to be the faulty GPU.

[0049] Step 203: Summarize the first diagnostic results corresponding to the channels belonging to the same communication group, and determine the second diagnostic result corresponding to the communication group level;

[0050] Based on step 202, this step aims to have the execution subject summarize the first diagnostic results corresponding to each channel belonging to the same communication group and determine the second diagnostic result corresponding to the communication group level. Specifically, the summarization method can include multiple methods such as finding the intersection method.

[0051] Step 204: Summarize the second diagnostic results corresponding to the communication groups belonging to the same training task, and determine the target diagnostic result corresponding to the training task level;

[0052] Based on step 203, this step aims to have the execution subject summarize the second diagnosis results corresponding to each communication group belonging to the same training task and determine the target diagnosis result corresponding to the training task level. Specifically, the summarization method can include multiple methods such as finding the intersection method.

[0053] It should be understood that steps 203 and 204 are intended to determine, through a summary approach, which GPUs are truly faulty and which are non-faulty GPUs that are mistakenly determined to be faulty GPUs due to the influence of truly faulty GPUs, that is, to eliminate incorrect answers through a summary approach.

[0054] Step 205: Determine the faulty GPU that affects the normal execution of the current training task based on the target diagnosis result.

[0055] Based on step 204, this step aims to determine, by the execution subject, the faulty GPU that affects the normal execution of the current training task based on the target diagnosis result. For example, the faulty GPU included in the target diagnosis result is directly determined as the faulty GPU.

[0056] The method for determining a faulty graphics processing chip in a computing cluster provided by the embodiments of the present disclosure obtains the number of collective communication operations executed, the channel number to which it belongs, and the execution status information of the currently executed collective communication operation for each GPU when executing the collective communication operation, and performs a multi-level and multi-dimensional comprehensive analysis on the obtained information related to the executed collective communication operations of each GPU. Finally, the faulty GPU that causes the current training task to hang is determined, thereby improving the efficiency of troubleshooting faulty nodes in a multi-node computing cluster, minimizing the duration of training tasks hanging due to faults, and ultimately reducing the overall time consumed in executing a single training task.

[0057] Please refer to Figure 3 , Figure 3 A flowchart of a method for diagnosing a faulty GPU at the channel level is provided in an embodiment of the present disclosure, wherein process 300 includes the following steps:

[0058] Step 301: For each GPU with the same channel number, obtain the corresponding executed collective communication operation count and execution status information;

[0059] Step 302: Determine whether the acquired executed aggregate communication operands are identical and whether the execution status information is all completed, and select different processing branches to execute according to different situations;

[0060] Step 303: Determine the GPU with the smallest number of executed collective communication operations as the first diagnosis result corresponding to the current channel number at the channel level;

[0061] This step is based on the judgment result of step 302 that the number of executed collective communication operations is not exactly the same and regardless of whether the execution status information of each execution is completed, and is intended to be determined by the above-mentioned execution subject as the first diagnostic result corresponding to the current channel number at the channel level.

[0062] Step 304: determining that no faults have occurred as a first diagnosis result corresponding to the current channel number at the channel level;

[0063] This step is based on the judgment result of step 302 that the number of executed collective communication operations is exactly the same and the execution status information of each is completed. It is intended for the above-mentioned execution subject to determine that no faults have occurred as the first diagnostic result corresponding to the current channel number at the channel level, that is, no faults have occurred in any GPU under the current channel number.

[0064] Step 305: Obtain the data size set by the user for each GPU with the same channel number;

[0065] This step is based on the judgment result of step 302 that the number of executed collective communication operations is exactly the same and the incomplete execution status information is all completed. The purpose of this step is to obtain the data size (i.e., Message Size) set by the user for each GPU with the same channel number by the above-mentioned execution subject.

[0066] Step 306: Determine whether the sizes of the acquired data are exactly the same. If not, execute step 307; otherwise, execute step 308.

[0067] Step 307: Outputting a third fault diagnosis indicating that the user has set an incorrect data size for different GPUs in the same channel;

[0068] This step is based on the judgment result of step 306 that the sizes of the acquired data are not exactly the same. It is intended for the above-mentioned execution subject to output a third fault diagnosis that the user has set an incorrect data size for different GPUs in the same channel, that is, there is a fault in the upper-level software that issues the training task, and a prompt or notification can be returned to suggest checking whether there is any abnormality in the upper-level software logic.

[0069] Step 308: Determine the actual operation type of the collective communication operation currently executed by each GPU having the same channel number;

[0070] This step is based on the judgment result of step 306 that the sizes of the acquired data are exactly the same, and is intended for the execution subject to determine the actual operation type of the collective communication operation currently executed by each GPU with the same channel number.

[0071] Among them, the actual operation types can include two categories, namely, asymmetric collective communication operations represented by Reduce (Chinese translation: aggregation) and Broadcast (broadcast), and symmetric collective communication operations represented by All Reduce (Chinese translation: full protocol or broadcast protocol), All Gather (Chinese translation: full collection), and Reduce Scatter (Chinese translation: scattered collection or aggregation distribution).

[0072] The characteristic of asymmetric collective communication operations is that all data chunks to be processed start from a certain GPU and are sent unidirectionally in a fixed sending order until the last GPU receives it. This can be called a one-way chain. The characteristic of symmetric collective communication operations is that all GPUs may be the source of a certain data chunk to be processed until it is received by the previous GPU of the current GPU. This can be relatively called a two-way chain.

[0073] Step 309: Obtain the number of processed data blocks of each GPU having the same channel number;

[0074] This step aims to obtain the number of processed data blocks of each GPU having the same channel number by the execution subject.

[0075] Step 310 : Based on the acquired number of processed data blocks and the actual operation type, determine a first diagnosis result corresponding to the current channel number at the channel level.

[0076] On the basis of step 308 and step 309 , this step aims to determine, by the execution subject, a first diagnostic result corresponding to the current channel number at the channel level based on the acquired number of processed data blocks and the actual operation type.

[0077] like Figure 4-1 Two specific cases are shown:

[0078] Case 1: When the actual operation type is a symmetric collective communication operation, the first GPU with the smallest number of processed data blocks and the preceding GPU of the first GPU are jointly determined as the first diagnosis result corresponding to the current channel number at the channel level.

[0079] Since in symmetric collective communication operations, all GPUs in the chain may be the source of a certain data block to be processed until it is received by the predecessor GPU of the current GPU, when a hang failure occurs, the number of processed data blocks of each GPU (which can be referred to by a counter) will have the following value: Figure 4-2 Features shown:

[0080] That is, the counter size of the faulty GPU must be at the bottom of the chain. Therefore, the GPU with the smallest counter and its predecessor GPU should be the final set of faulty GPUs in the current channel.

[0081] Case 2: When the actual operation type is specifically an asymmetric collective communication operation, the second GPU that first has the smallest number of processed data blocks in the data flow and the predecessor GPU of the second GPU are jointly determined as the first diagnosis result corresponding to the current channel number at the channel level.

[0082] In symmetric collective communication operations, all chunks start from a certain GPU and are sent unidirectionally in a fixed order until the last GPU receives them. Therefore, when a hang occurs, the data sent by the preceding GPU of the failed link to the failed GPU cannot be received. Therefore, its counter characteristics are as follows: Figure 4-3 As shown in the figure: It decreases monotonically starting from the data sending source GPU and remains unchanged after the faulty GPU. Therefore, the GPU with the smallest counter and its predecessor GPU should be the final faulty GPU set of the current channel.

[0083] This embodiment provides faulty GPU diagnostic solutions for a variety of different situations at the channel level through steps 301-310. The diagnostic solution for each situation included therein can independently form an independent embodiment with the embodiment provided by process 200. There is no causal or dependent relationship between the diagnostic solutions in each situation. This embodiment only exists as a preferred embodiment that simultaneously embodies the diagnostic solutions for various situations.

[0084] Based on any of the above examples, see Figure 5 , Figure 5 A flowchart of a method for summarizing and eliminating erroneous diagnosis results by obtaining intersections is provided in an embodiment of the present disclosure. The process 500 includes the following steps:

[0085] Step 501: Obtain an intersection of first diagnosis results corresponding to different channels belonging to the same communication group to obtain a first intersection result;

[0086] Step 502: Determine the GPU recorded in the first intersection result as the second diagnosis result corresponding to the current communication group at the communication group level;

[0087] Step 503: Calculate the intersection of the second diagnosis results corresponding to different communication groups belonging to the same training task to obtain a second intersection result;

[0088] Step 504: Determine the GPU recorded in the second intersection result as the target diagnosis result corresponding to the current training task at the training task level;

[0089] Step 505: Determine the GPU recorded in the target diagnosis result as a faulty GPU that affects the normal execution of the current training task.

[0090] That is, this embodiment provides a final determination solution for the faulty GPU by eliminating incorrect answers by obtaining intersections for different channels under the communication group level and different communication groups under the training task level through steps 501 to 505. That is, only the faulty GPU that is the result of the intersection is the real faulty GPU.

[0091] Based on any of the above examples, see Figure 6 , Figure 6 This is a flowchart of a method for repairing a faulty GPU provided by an embodiment of the present disclosure. The process 600 includes the following steps:

[0092] Step 601: diagnose the fault type of the faulty GPU and determine the target fault type;

[0093] Step 602: Repair the faulty GPU according to the fault repair method corresponding to the target fault type, and obtain a fault repair result;

[0094] All feasible or operable fault repair methods can be configured in advance for different known fault types, such as automatic repair methods, manual repair methods, or fault repair methods such as replacing a GPU.

[0095] Step 603: In response to the fault repair result being that the fault has been repaired, continue to execute the current training task.

[0096] This embodiment aims to attempt to repair the faulty GPU located according to the solution provided in the above embodiment, so as to eliminate the hanging problem as soon as possible through the fault repair method and continue to execute the current training task.

[0097] To deepen understanding, this disclosure also provides a specific implementation solution based on a specific application scenario. The overall architecture design can be found in Figure 7-1 As shown, Figure 7-1 middle:

[0098] 1. Communication status profiling (data collection, information gathering): When the training task is running normally, the AI ​​Communication Library (also known as the AI ​​Collective Communication Library, CCL) collects statistics on the collective communication status in real time;

[0099] 2. Fault status output: When the fault detection process finds that the training task is hung, it notifies the training task process to print the collective communication status of all communication groups at the time of the hang through Unix socket (Chinese name: Unix domain socket) or other inter-process communication methods;

[0100] 3. Hang Node Analysis: The training task fault diagnosis controller collects all TraceHang logs for the task, analyzes the faulty nodes, and outputs them to the user.

[0101] To support the above diagnostic architecture, the following needs to be implemented:

[0102] 1. Collective Communication Library (CCL): Responsible for real-time statistics of collective communication status information, and printing the collective communication status information of the failure when the training task fails:

[0103] 1. When the communication group is initialized, a block of UM memory (shared memory) is allocated for each channel of each rank (i.e., a uniquely numbered GPU) in each communication group to record the TraceHang counter. This memory is mapped to video memory and memory using logic similar to the MMAP (Memory Mapped Access Protocol, used to map a file or other object into memory), making it accessible to both the GPU and the CPU.

[0104] 2. When the application layer submits each collective communication operation, it submits the TraceHang counter as a parameter to the GPU through launchKernel (‌, an important concept in GPU programming, used to start kernel execution on the GPU) and records information such as the message size (the amount of data described above), opCount (the number of collective communication operations executed described above), and channel (the channel number described above).

[0105] 3. When executing the collective communication algorithm on the GPU side, use the TraceHang counter to record the number of chunks processed in the receiving and sending directions of the current block (corresponding channel) when processing the current collective communication operation;

[0106] 4. Mark the status of the current collective communication operation on the GPU side, such as: init (not started), processing (in progress), done (execution completed);

[0107] 5. After the training task hangs, the AI ​​communication library will receive the external Unix socket signal and print the TraceHang counter of all ranks of all communication groups to the log file on the CPU side.

[0108] 2. Training task fault diagnosis controller:

[0109] 1. Monitor the running status of training tasks and find out if they are hung in time;

[0110] 2. When a training task is found to be hung, a unixsocket signal is sent to the training task processes of all worker nodes to trigger the communication library log printing;

[0111] 3. After the communication library finishes printing the log, collect the training task logs of all nodes;

[0112] 4. Conduct centralized diagnosis and analysis based on the fault pattern.

[0113] 3. Core fault diagnosis logic:

[0114] Step 1: Aggregate all data for a single task by communication group ID. The diagnostic logic for a single communication group is detailed in step 2. After the diagnosis of a single communication group is completed, aggregate the diagnostic results for all communication groups, as detailed in step 6.

[0115] Step 2: Aggregate all data from the same communication group by channel ID. See Step 3 for details on the diagnostic logic for a single channel. After the diagnosis of a single channel is complete, aggregate the diagnostic results for all channels in the communication group. See Step 5 for details.

[0116] Step 3: For the data of the same channel, the diagnostic process can be found in Figure 3 ,The chain diagnosis process is detailed in step 4. If the diagnosis result is a software fault, then the fault is the root cause of the training task hang failure;

[0117] Step 4: During initialization, the communication library performs a channel search process. Each channel searches for a communication link and submits it to the GPU. All collective communication operations can be roughly divided into two categories: asymmetric collective communication operations and symmetric collective communication operations. The faulty GPU cluster can be accurately determined based on the different types of collective communication operations and the count of executed data blocks.

[0118] Step 5: After confirming the fault rank of a single channel, the results need to be summarized for all channels in a single communication group. According to the principle of hang failure, if a certain rank is the root cause of the failure, all faulty channels in the communication group will point to that rank. Therefore, the intersection of the fault ranks of all faulty channels in the communication group needs to be taken, which is the faulty node of the communication group.

[0119] Step 6: After confirming the failure rank of a single communication group, it is necessary to summarize the results of all communication groups of the training task. Training tasks usually use complex hybrid parallel logic, such as tensor parallelism, pipeline parallelism, data parallelism, etc. Figure 7-2As shown in the figure below. According to the TraceHang fault principle, if a GPU is the root cause of the fault, all communication groups containing that GPU will be directed to that GPU. Therefore, the diagnostic results for each communication group must be filtered according to the above principle. The resulting result is the faulty GPU node determined by TraceHang.

[0120] For ease of understanding, the following description will be given with reference to specific examples:

[0121] 1. Channel diagnosis

[0122] 1.1 Asymmetric collective communication operations

[0123] The data flow is from rank 0 to rank 1, and then to rank 7. The counter characteristic along this data flow is: it first decreases monotonically and then remains constant. Assuming the counter for a single channel is shown in Table 1, rank 3 and the node preceding rank 3 are the faulty ranks. The diagnostic result is: rank 3, rank 4.

[0124] Table 1

[0125]

[0126] 1.2 Symmetric collective communication operations

[0127] The data flow is: rank 0 → rank 1 → .... rank 7 → rank 0. The counter characteristic along the data flow is: first monotonically decreasing, then monotonically increasing. Assuming the counter for a single channel is as shown in Table 2 below, the trough node and its preceding node are faulty nodes. The trough node is rank 1. Therefore, the faulty nodes diagnosed are: rank 0, rank 1.

[0128] Table 2

[0129]

[0130] 2. Comm (communication group) diagnosis: multi-channel aggregation

[0131] The principle of aggregating multi-channel diagnosis results is: if rank A is a faulty node, then rank A will definitely be diagnosed as a faulty node in any faulty channel.

[0132] Assume that the diagnosis of multiple channels of a single communication group is summarized in Table 3:

[0133] Table 3

[0134]

[0135] 1) Rank 7 is not a faulty node because it has not been diagnosed as a faulty node in channel 1;

[0136] 2) Rank 8 is a faulty node because it is diagnosed as a faulty node in all faulty channels (channel 0 and channel 1);

[0137] 3) Rank 9 is not a faulty node because it has not been diagnosed as a faulty node in channel 0;

[0138] 4) In summary: The faulty node diagnosed by this comm is rank 8.

[0139] 3. Task diagnosis: multi-comm aggregation

[0140] The principle of aggregating multi-comm diagnosis results is: if rank A is a faulty node, then rank A will definitely be diagnosed as a faulty node in all faulty comms that contain this node.

[0141] Assume that the diagnosis results of multiple communication groups for a single training task are shown in Table 4 below:

[0142] Table 4

[0143]

[0144] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for determining a faulty graphics processing chip in a computing cluster. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0145] like Figure 8As shown, the device 800 for determining a faulty graphics processing chip in a computing power cluster of this embodiment may include: an information acquisition unit 801, a channel-level diagnostic result determination unit 802, a communication group-level diagnostic result determination unit 803, a training task-level diagnostic result determination unit 804, and a faulty GPU positioning unit 805. Among them, the information acquisition unit 801 is configured to respectively obtain the number of executed collective communication operations, channel number and execution status information of the currently executed collective communication operation corresponding to each graphics processing chip GPU constituting the computing power cluster; the channel-level diagnostic result determination unit 802 is configured to determine the first diagnostic result corresponding to the channel level for each GPU with the same channel number based on the corresponding number of executed collective communication operations and execution status information; the communication group-level diagnostic result determination unit 803 is configured to summarize the first diagnostic results corresponding to each channel belonging to the same communication group, and determine the second diagnostic result corresponding to the communication group level; the training task-level diagnostic result determination unit 804 is configured to summarize the second diagnostic results corresponding to each communication group belonging to the same training task, and determine the target diagnostic result corresponding to the training task level; the faulty GPU positioning unit 805 is configured to determine the faulty GPU that affects the normal execution of the current training task based on the target diagnostic result.

[0146] In this embodiment, the specific processing of the information acquisition unit 801, the channel level diagnosis result determination unit 802, the communication group level diagnosis result determination unit 803, the training task level diagnosis result determination unit 804 and the faulty GPU positioning unit 805 and the technical effects thereof can be referred to respectively. Figure 2 The relevant descriptions of steps 201-205 in the corresponding embodiment are not repeated here.

[0147] In some optional implementations of this embodiment, the channel-level diagnostic result determining unit 802 may include:

[0148] The executed operation count and execution status information acquisition subunit is configured to acquire the corresponding executed collective communication operation count and execution status information for each GPU having the same channel number;

[0149] The first diagnosis processing subunit is configured to, in response to the obtained executed collective communication operation numbers being different, determine the GPU with the smallest executed collective communication operation number as the first diagnosis result corresponding to the current channel number at the channel level.

[0150] In some optional implementations of this embodiment, the channel-level diagnostic result determining unit 802 may further include:

[0151] The second diagnosis processing sub-unit is configured to, in response to the obtained executed collective communication operation numbers being exactly the same and the execution status information being all completed, determine that no faults have occurred as a first diagnosis result corresponding to the current channel number at the channel level.

[0152] In some optional implementations of this embodiment, the channel-level diagnostic result determining unit 802 may further include:

[0153] The data size acquisition subunit is configured to, in response to the acquired executed aggregate communication operation numbers being identical and the respective execution status information being incomplete, acquire the data size set by the user for each GPU having the same channel number;

[0154] The third diagnosis processing subunit is configured to output a third fault diagnosis that the user has set an incorrect data size for different GPUs in the same channel in response to the acquired data sizes not being completely the same.

[0155] In some optional implementations of this embodiment, the channel-level diagnostic result determining unit 802 may further include:

[0156] an operation type determination subunit configured to, in response to the obtained data amounts being identical in size, determine an actual operation type of the collective communication operation currently executed by each GPU having the same channel number; wherein the actual operation type includes a symmetric operation or an asymmetric operation;

[0157] a processed data block acquisition subunit, configured to acquire the number of processed data blocks of each GPU having the same channel number;

[0158] The fourth diagnosis processing subunit is configured to determine a first diagnosis result corresponding to a current channel number at the channel level based on the acquired number of processed data blocks and the actual operation type.

[0159] In some optional implementations of this embodiment, the fourth diagnosis processing subunit may include:

[0160] The fifth diagnosis module is configured to, in response to the actual operation type being a symmetric operation, jointly determine the first GPU having the smallest number of processed data blocks and the preceding GPU of the first GPU as the first diagnosis result corresponding to the current channel number at the channel level.

[0161] In some optional implementations of this embodiment, the fourth diagnosis processing subunit may include:

[0162] The sixth diagnostic module is configured to, in response to the actual operation type being an asymmetric operation, jointly determine the second GPU that first has the smallest number of processed data blocks in the data flow and the GPU preceding the second GPU as the first diagnostic result corresponding to the current channel number at the channel level.

[0163] In some optional implementations of this embodiment, the information acquisition unit 801 may be further configured to:

[0164] From the shared memory allocated in advance for each GPU, the number of collective communication operations executed, the channel number, and the execution status information of the currently executed collective communication operation corresponding to each GPU are obtained respectively.

[0165] In some optional implementations of this embodiment, the communication group level diagnosis result determining unit 803 may be further configured to:

[0166] Obtaining an intersection of first diagnosis results corresponding to different channels belonging to the same communication group to obtain a first intersection result;

[0167] The GPU recorded in the first intersection result is determined as the second diagnosis result corresponding to the current communication group at the communication group level.

[0168] In some optional implementations of this embodiment, the training task level diagnosis result determination unit 804 may be further configured to:

[0169] Obtaining an intersection of the second diagnosis results corresponding to different communication groups belonging to the same training task to obtain a second intersection result;

[0170] The GPU recorded in the second intersection result is determined as the target diagnosis result corresponding to the current training task at the training task level.

[0171] In some optional implementations of this embodiment, the faulty GPU locating unit 805 may be further configured to:

[0172] The GPU recorded in the target diagnosis result is determined to be a faulty GPU that affects the normal execution of the current training task.

[0173] In some optional implementations of this embodiment, the apparatus 800 for determining a faulty graphics processing chip in a computing power cluster may further include:

[0174] a fault type diagnosis unit configured to perform fault type diagnosis on the faulty GPU and determine a target fault type;

[0175] a fault repair unit configured to perform fault repair on the faulty GPU according to a fault repair method corresponding to a target fault type, and obtain a fault repair result;

[0176] The training task continuing execution unit is configured to continue executing the current training task in response to the fault repair result being that the fault has been repaired.

[0177] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The apparatus for determining a faulty graphics processing chip in a computing power cluster provided by this embodiment obtains the number of collective communication operations executed, the channel number to which it belongs, and the execution status information of the currently executed collective communication operation for each GPU in a computing power cluster composed of multiple GPUs, and performs a multi-level and multi-dimensional comprehensive analysis on the obtained information related to the executed collective communication operations of each GPU, and finally determines the faulty GPU that causes the current training task to hang, thereby improving the efficiency of troubleshooting faulty nodes in multi-node computing power clusters, minimizing the duration of training tasks hanging due to faults, and ultimately reducing the overall time taken to execute a single training task.

[0178] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that when the at least one processor executes, it can implement the method for determining a faulty graphics processing chip in a computing power cluster described in any of the above embodiments.

[0179] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the method for determining a faulty graphics processing chip in a computing power cluster described in any of the above embodiments when executed.

[0180] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, which, when executed by a processor, can implement the method for determining a faulty graphics processing chip in a computing power cluster described in any of the above embodiments.

[0181] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0182] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. Computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0183] Various components in device 600 are connected to I / O interface 605, including an input unit 606, such as a keyboard, mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, optical disk, etc.; and a communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0184] The computing unit 601 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the method for determining a faulty graphics processing chip within a computing power cluster. For example, in some embodiments, the method for determining a faulty graphics processing chip within a computing power cluster can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method for determining a faulty graphics processing chip within a computing power cluster described above can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured in any other appropriate manner (eg, by means of firmware) to execute the method for determining a faulty graphics processing chip in a computing power cluster.

[0185] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0186] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0187] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0188] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0189] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0190] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers and establishing a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host. This is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and virtual private server (VPS) services.

[0191] According to the technical solution of the embodiment of the present disclosure, for a computing power cluster composed of multiple GPUs, by obtaining the number of collective communication operations executed, the channel number to which they belong, and the execution status information of the currently executed collective communication operation of each GPU when executing the collective communication operation, and performing a multi-level and multi-dimensional comprehensive analysis of the information related to the executed collective communication operations of each GPU, the faulty GPU that caused the current training task to hang can be finally determined, thereby improving the efficiency of troubleshooting faulty nodes in multi-node computing power clusters, minimizing the duration of training tasks hanging due to faults, and ultimately reducing the overall time consumed in executing a single training task.

[0192] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0193] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for determining a faulty graphics processing chip in a computing cluster, comprising: Respectively obtain the number of executed collective communication operations, the channel number, and the execution status information of the currently executed collective communication operation corresponding to each graphics processing chip GPU constituting the computing power cluster; For each GPU having the same channel number, determining a first diagnostic result corresponding to the channel level based on the corresponding executed collective communication operations and the execution state information; Summarize the first diagnostic results corresponding to the channels belonging to the same communication group, and determine the second diagnostic result corresponding to the communication group level; Summarize the second diagnostic results corresponding to each communication group belonging to the same training task, and determine the target diagnostic result corresponding to the training task level; A faulty GPU that affects the normal execution of the current training task is determined based on the target diagnosis result.

2. The method according to claim 1, wherein The step of determining, for each GPU having the same channel number, a first diagnostic result corresponding to a channel level based on a corresponding number of executed collective communication operations and execution state information, includes: For each GPU with the same channel number, the corresponding executed collective communication operation count and execution status information are obtained; In response to the acquired numbers of executed collective communication operations being different from each other, a GPU having the smallest number of executed collective communication operations is determined as a first diagnosis result corresponding to a current channel number at the channel level.

3. The method according to claim 2, further comprising: In response to the acquired executed aggregate communication operation numbers being identical and the execution status information indicating that the execution is complete, it is determined that no faults have occurred as a first diagnosis result corresponding to the current channel number at the channel level.

4. The method according to claim 2, further comprising: In response to the acquired executed collective communication operation numbers being identical and the execution status information of each being incomplete indicating execution completion, acquiring a data size set by a user for each GPU having the same channel number; In response to the acquired data sizes not being completely the same, a third fault diagnosis is outputted, indicating that the user has set an incorrect data size for different GPUs in the same channel.

5. The method according to claim 4, further comprising: In response to the obtained data amounts being exactly the same, determining an actual operation type of the collective communication operation currently executed by each GPU having the same channel number; wherein the actual operation type includes a symmetric operation or an asymmetric operation; Get the number of processed data blocks of each GPU with the same channel number; Based on the acquired number of processed data blocks and the actual operation type, a first diagnostic result corresponding to the current channel number at the channel level is determined.

6. The method according to claim 5, wherein: The determining, based on the obtained number of processed data blocks and the actual operation type, a first diagnostic result corresponding to the current channel number at the channel level includes: In response to the actual operation type being the symmetric operation, a first GPU having the smallest number of processed data blocks and a preceding GPU of the first GPU are jointly determined as a first diagnosis result corresponding to a current channel number at the channel level.

7. The method according to claim 5, wherein: The determining, based on the obtained number of processed data blocks and the actual operation type, a first diagnostic result corresponding to the current channel number at the channel level includes: In response to the actual operation type being the asymmetric operation, the second GPU that first has the smallest number of processed data blocks in the data flow and the preceding GPU of the second GPU are jointly determined as the first diagnosis result corresponding to the current channel number at the channel level.

8. The method according to claim 1, wherein The step of respectively obtaining the number of executed collective communication operations, the channel number, and the execution status information of the currently executed collective communication operation corresponding to each graphics processing chip GPU constituting the computing power cluster includes: The number of executed collective communication operations, the channel number, and the execution status information of the currently executed collective communication operation corresponding to each GPU are respectively obtained from the shared memory allocated in advance to each GPU.

9. The method according to any one of claims 1 to 8, wherein: The aggregating the first diagnostic results corresponding to the channels belonging to the same communication group and determining the second diagnostic result corresponding to the communication group level includes: Obtaining an intersection of first diagnosis results corresponding to different channels belonging to the same communication group to obtain a first intersection result; The GPU recorded in the first intersection result is determined as a second diagnosis result corresponding to the current communication group at the communication group level.

10. The method according to claim 9, wherein: The summarizing of the second diagnostic results corresponding to the communication groups belonging to the same training task and determining the target diagnostic result corresponding to the training task level includes: Obtaining an intersection of the second diagnosis results corresponding to different communication groups belonging to the same training task to obtain a second intersection result; The GPU recorded in the second intersection result is determined as the target diagnosis result corresponding to the current training task at the training task level.

11. The method according to claim 10, wherein: The determining, based on the target diagnosis result, a faulty GPU that affects normal execution of the current training task includes: The GPU recorded in the target diagnosis result is determined as a faulty GPU that affects the normal execution of the current training task.

12. The method according to claim 11, further comprising: Performing fault type diagnosis on the faulty GPU to determine a target fault type; Performing fault repair on the faulty GPU according to a fault repair method corresponding to the target fault type to obtain a fault repair result; In response to the fault repair result being that the fault has been repaired, the current training task continues to be executed.

13. A device for determining a faulty graphics processing chip in a computing cluster, comprising: An information acquisition unit is configured to respectively acquire the number of executed collective communication operations, the channel number, and the execution status information of the currently executed collective communication operation corresponding to each graphics processing chip GPU constituting the computing power cluster; a channel-level diagnostic result determination unit configured to determine, for each GPU having the same channel number, a first diagnostic result corresponding to the channel level based on the corresponding executed aggregate communication operation count and execution state information; a communication group level diagnosis result determination unit configured to aggregate first diagnosis results corresponding to channels belonging to the same communication group and determine a second diagnosis result corresponding to the communication group level; a training task level diagnosis result determination unit configured to aggregate the second diagnosis results corresponding to the communication groups belonging to the same training task and determine a target diagnosis result corresponding to the training task level; The faulty GPU locating unit is configured to determine the faulty GPU that affects the normal execution of the current training task based on the target diagnosis result.

14. The device according to claim 13, wherein The channel-level diagnostic result determination unit includes: The executed operation count and execution status information acquisition subunit is configured to acquire the corresponding executed collective communication operation count and execution status information for each GPU having the same channel number; The first diagnosis processing subunit is configured to, in response to the obtained executed collective communication operation numbers being different, determine the GPU with the smallest executed collective communication operation number as the first diagnosis result corresponding to the current channel number at the channel level.

15. The device according to claim 14, wherein The channel level diagnostic result determination unit further includes: The second diagnosis processing sub-unit is configured to, in response to the obtained executed collective communication operation numbers being exactly the same and the execution status information being all completed, determine that no faults have occurred as a first diagnosis result corresponding to the current channel number at the channel level.

16. The device according to claim 14, wherein The channel level diagnostic result determination unit further includes: The data size acquisition subunit is configured to, in response to the acquired executed aggregate communication operation numbers being identical and the respective execution status information being incomplete, acquire the data size set by the user for each GPU having the same channel number; The third diagnosis processing subunit is configured to output a third fault diagnosis that the user has set an incorrect data size for different GPUs in the same channel in response to the acquired data sizes not being completely the same.

17. The device according to claim 16, wherein The channel level diagnostic result determination unit further includes: an operation type determination subunit configured to, in response to the acquired data amounts being identical in size, determine an actual operation type of the collective communication operation currently executed by each GPU having the same channel number; wherein the actual operation type includes a symmetric operation or an asymmetric operation; a processed data block acquisition subunit, configured to acquire the number of processed data blocks of each GPU having the same channel number; The fourth diagnosis processing subunit is configured to determine a first diagnosis result corresponding to a current channel number at the channel level based on the acquired number of processed data blocks and the actual operation type.

18. The device according to claim 17, wherein The fourth diagnosis processing subunit includes: The fifth diagnostic module is configured to, in response to the actual operation type being the symmetric operation, jointly determine the first GPU having the smallest number of processed data blocks and the predecessor GPU of the first GPU as the first diagnostic result corresponding to the current channel number at the channel level.

19. The device according to claim 17, wherein The fourth diagnosis processing subunit includes: The sixth diagnostic module is configured to, in response to the actual operation type being the asymmetric operation, jointly determine the second GPU that first has the smallest number of processed data blocks in the data flow and the preceding GPU of the second GPU as the first diagnostic result corresponding to the current channel number at the channel level.

20. The apparatus according to claim 13, wherein The information acquisition unit is further configured to: The number of executed collective communication operations, the channel number, and the execution status information of the currently executed collective communication operation corresponding to each GPU are respectively obtained from the shared memory allocated in advance to each GPU.

21. The device according to any one of claims 13 to 20, wherein: The communication group level diagnosis result determination unit is further configured to: Obtaining an intersection of first diagnosis results corresponding to different channels belonging to the same communication group to obtain a first intersection result; The GPU recorded in the first intersection result is determined as a second diagnosis result corresponding to the current communication group at the communication group level.

22. The device according to claim 21, wherein The training task level diagnosis result determination unit is further configured to: Obtaining an intersection of the second diagnosis results corresponding to different communication groups belonging to the same training task to obtain a second intersection result; The GPU recorded in the second intersection result is determined as the target diagnosis result corresponding to the current training task at the training task level.

23. The device according to claim 22, wherein The faulty GPU locating unit is further configured to: The GPU recorded in the target diagnosis result is determined as a faulty GPU that affects the normal execution of the current training task.

24. The apparatus according to claim 23, further comprising: a fault type diagnosis unit configured to perform fault type diagnosis on the faulty GPU and determine a target fault type; a fault repair unit configured to perform fault repair on the faulty GPU according to a fault repair method corresponding to the target fault type, and obtain a fault repair result; The training task continuing execution unit is configured to continue executing the current training task in response to the fault repair result being that the fault has been repaired.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for determining a faulty graphics processing chip in a computing power cluster according to any one of claims 1-12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method for determining a faulty graphics processing chip in a computing power cluster according to any one of claims 1-12.

27. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method for determining a faulty graphics processing chip in a computing power cluster according to any one of claims 1 to 12.

Images