Method and device for determining fault graphic processing chip in computing power cluster and electronic equipment

Abstract

The invention provides a method and device for determining a fault graphic processing chip in a computing power cluster and electronic equipment, and relates to the technical field of generative large language models, graphic processing chips, computing power clusters, ensemble communication operation and the like. The method comprises the following steps: respectively acquiring executed set communication operands, channel numbers and execution state information of currently executed set communication operation corresponding to each GPU forming the computing power cluster; determining a first diagnosis result for each GPU with the same channel number based on the corresponding executed set communication operands and the execution state information; summarizing the first diagnosis results corresponding to the channels belonging to the same communication group, and determining a second diagnosis result; summarizing second diagnosis results corresponding to the communication groups belonging to the same training task, and determining a target diagnosis result; and determining a faulted GPU influencing normal execution of the current training task based on the target diagnosis result. According to the method, the faulted GPU causing the training task hang can be accurately positioned.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, specifically to the field of artificial intelligence technology such as generative large language models, graphics processing chips, computing power clusters, collective communication operations, and more particularly to a method, device, electronic device, computer-readable storage medium, and computer program product for determining a faulty graphics processing chip in a computing power cluster. Background Art

[0002] In large model training, due to the significant increase in model size and data volume, multiple computing nodes (a GPU usually exists as a computing node) need to work together to complete the training task. To achieve efficient data sharing and updating between nodes, computing nodes usually use the Collective Communication Library (CCL) to perform collective communication operations, such as broadcast and all-reduce, to ensure that each node participating in the training can complete the parameter update at each training step, thereby maintaining the consistency of the model.

[0003] However, the synchronization mechanism of collective communication also requires coordination among all nodes during the training process. Any abnormality in a single node during the training of a large model will affect the progress of the overall task. In particular, if a single node is blocked or "hangs" (which can be understood as hanging or getting stuck and no longer needing to be executed), all other nodes will also be forced to "hang" while waiting for synchronization data, causing the entire training task to be unable to continue. However, such problems lack explicit features and it is difficult to quickly locate faulty nodes, especially in a multi-node cluster environment. The fault point of a hung node is not easy to identify, which further increases the difficulty of troubleshooting and seriously affects training efficiency and system stability. Summary of the invention

[0004] The embodiments of the present disclosure provide a method, device, electronic device, computer-readable storage medium, and computer program product for determining a faulty graphics processing chip in a computing power cluster.

[0005] In the first aspect, an embodiment of the present disclosure proposes a method for determining a faulty graphics processing chip in a computing power cluster, including: respectively obtaining the number of executed collective communication operations, the channel number, and the execution status information of the currently executed collective communication operation corresponding to each graphics processing chip GPU constituting the computing power cluster; for each GPU with the same channel number, based on the corresponding number of executed collective communication operations and the execution status information, determining a first diagnostic result corresponding to the channel level; summarizing the first diagnostic results corresponding to each channel belonging to the same communication group, and determining a second diagnostic result corresponding to the communication group level; summarizing the second diagnostic results corresponding to each communication group belonging to the same training task, and determining a target diagnostic result corresponding to the training task level; determining the faulty GPU that affects the normal execution of the current training task based on the target diagnostic result.

[0006] In the second aspect, an embodiment of the present disclosure proposes a device for determining a faulty graphics processing chip in a computing power cluster, comprising: an information acquisition unit, configured to respectively acquire the number of executed collective communication operations, channel numbers and execution status information of the currently executed collective communication operations corresponding to each graphics processing chip GPU constituting the computing power cluster; a channel-level diagnostic result determination unit, configured to determine, for each GPU having the same channel number, a first diagnostic result corresponding to the channel level based on the corresponding number of executed collective communication operations and execution status information; a communication group-level diagnostic result determination unit, configured to summarize the first diagnostic results corresponding to each channel belonging to the same communication group, and determine the second diagnostic result corresponding to the communication group level; a training task-level diagnostic result determination unit, configured to summarize the second diagnostic results corresponding to each communication group belonging to the same training task, and determine the target diagnostic result corresponding to the training task level; a faulty GPU locating unit, configured to determine the faulty GPU that affects the normal execution of the current training task based on the target diagnostic result.

[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that when the at least one processor executes, the method for determining a faulty graphics processing chip within a computing power cluster as described in the first aspect can be implemented.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the method for determining a faulty graphics processing chip in a computing power cluster as described in the first aspect when executed.

[0009] In a fifth aspect, an embodiment of the present disclosure provides a computer program product including a computer program, which, when executed by a processor, can implement the steps of the method for determining a faulty graphics processing chip in a computing power cluster as described in the first aspect.

[0010] The solution for determining the faulty graphics processing chip in a computing cluster provided by the present invention is to obtain the number of collective communication operations executed, the channel number to which they belong, and the execution status information of the currently executed collective communication operation of each GPU when executing the collective communication operation, and perform a multi-level and multi-dimensional comprehensive analysis on the obtained information related to the executed collective communication operations of each GPU, so as to finally determine the faulty GPU that causes the current training task to hang, thereby improving the efficiency of troubleshooting faulty nodes in a multi-node computing cluster, minimizing the duration of training tasks hanging due to faults, and ultimately reducing the overall time consumed in executing a single training task.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0013] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0014] Figure 2 A flowchart of a method for determining a faulty graphics processing chip in a computing cluster provided by an embodiment of the present disclosure;

[0015] Figure 3 A flow chart of a method for diagnosing a faulty GPU at the channel level provided in an embodiment of the present disclosure;

[0016] Figure 4-1 Two branch schematic diagrams for obtaining the first diagnosis results in different situations provided by the embodiment of the present disclosure;

[0017] Figure 4-2 A schematic diagram of counter features of each GPU corresponding to the symmetric combined communication operation provided in an embodiment of the present disclosure;

[0018] Figure 4-3 A schematic diagram of counter features of each GPU corresponding to an asymmetric combined communication operation provided in an embodiment of the present disclosure;

[0019] Figure 5 A flowchart of a method for summarizing and eliminating erroneous diagnosis results by obtaining intersections provided in an embodiment of the present disclosure;

[0020] Figure 6 A flowchart of a fault repair method for a faulty GPU provided by an embodiment of the present disclosure;

[0021] Figure 7-1 A structural flow chart of another system for determining a faulty graphics processing chip in a computing power cluster provided by an embodiment of the present disclosure;

[0022] Figure 7-2 A schematic diagram of the parallel structure between GPUs in a computing cluster provided by an embodiment of the present disclosure;

[0023] Figure 8 A structural block diagram of a device for determining a faulty graphics processing chip in a computing power cluster provided by an embodiment of the present disclosure;

[0024] Fig. 9 A schematic diagram of the structure of an electronic device suitable for executing a method for determining a faulty graphics processing chip in a computing power cluster provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.

[0026] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0027] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the method, apparatus, electronic device, and computer-readable storage medium for determining a faulty graphics processing chip in a computing cluster of the present disclosure may be applied.

[0028] like Figure 1 As shown, the system architecture 100 may include a server 110, a network 120, and a computing cluster 130. The computing cluster 130 includes multiple graphics processors, such as 131, 132, 133, 134, and 135 (not just 5, Figure 1Only a few numbers are used as examples), and the graphics cards 131, 132, 133, 134 and 135 include at least one graphics processing chip (i.e., GPU), that is, the computing power cluster 130 is a computing power cluster of multiple computing nodes; the network 120 is used to provide a medium for a communication link between the server 110 and the computing power cluster 130, and the network 120 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0029] The user can use the terminal device to interact with the server 110 through the network 120 to receive or send messages, etc. The server 110, the computing cluster 130 and the terminal device can be installed with various applications for realizing information communication between the two, such as fault GPU troubleshooting applications, training task delivery applications, instant messaging applications, etc.

[0030] The terminal device and the server 110 can be hardware or software. When the terminal device is hardware, it can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.; when the terminal device is software, it can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, without specific limitation here. When the server 110 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server; when the server 110 is software, it can be implemented as multiple software or software modules, or as a single software or software module, without specific limitation here. The graphics processor constituting the computing power cluster 130 is usually manifested as hardware. Of course, in special scenarios (such as simulation scenarios), it can also be manifested as software or software running products, which are not specifically limited here.

[0031] The server 110 can provide various services through various built-in applications. Taking a faulty GPU troubleshooting application that can provide a faulty GPU troubleshooting service as an example, the server 110 can achieve the following effects when running the faulty GPU troubleshooting application: first, the number of executed collective communication operations, channel number and execution status information of the currently executed collective communication operation corresponding to each GPU constituting the computing power cluster 130 are respectively obtained through the network 120; then, for each GPU with the same channel number, based on the corresponding number of executed collective communication operations and execution status information, a first diagnostic result corresponding to the channel level is determined; then, the first diagnostic results corresponding to each channel belonging to the same communication group are summarized, and a second diagnostic result corresponding to the communication group level is determined; next, the second diagnostic results corresponding to each communication group belonging to the same training task are summarized, and a target diagnostic result corresponding to the training task level is determined; finally, the faulty GPU that affects the normal execution of the current training task is determined based on the target diagnostic result.

[0032] Furthermore, after determining the faulty GPU, the server 110 may also send the information to the computing cluster 130 through the network 120, so that the computing cluster 130 may try to repair the faulty GPU using a feasible troubleshooting solution to resume the training task as soon as possible.

[0033] The methods for determining faulty graphics processing chips in a computing cluster provided in the subsequent embodiments of the present disclosure are generally performed by a server 110 having strong computing power and more computing resources. Accordingly, the training strategy determination device for a computing cluster that uses a mixture of chips of various models is generally also set in the server 110. Of course, it can also be performed by a device with similar computing power placed in or near the computing cluster 130.

[0034] It should be understood that Figure 1 The number of servers, networks, computing clusters, and graphics cards in the example is only illustrative. Any number of servers, networks, computing clusters, and graphics cards may be provided as required.

[0035] Please refer to Figure 2 , Figure 2 This is a flow chart of a method for determining a faulty graphics processing chip in a computing cluster provided by an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0036] Step 201: respectively obtaining the number of executed collective communication operations, the channel number, and the execution status information of the currently executed collective communication operation corresponding to each GPU constituting the computing power cluster;

[0037] This step is intended to determine the execution subject of the method by the faulty graphics processing chip in the computing power cluster (for example Figure 1 The server 110 shown in the figure obtains the number of executed collective communication operations, the channel number and the execution status information of the currently executed collective communication operation corresponding to each GPU constituting the computing power cluster.

[0038] The number of executed collective communication operations refers to the total number of communication operations that have been completed by the GPU during the execution process. These communication operations may include tasks such as data transmission, synchronization, and collaborative computing. The workload and performance of each GPU may be affected by the number of these operations. The number of operations executed can help evaluate the task progress and current load status of each GPU. In multi-GPU parallel computing systems, communication operations are usually an important part of the computing process, especially in large-scale parallel computing. Efficient communication between GPUs is the key to ensuring system performance. By tracking the number of communication operations executed by each GPU, computing resources can be better managed, and computing task scheduling and load balancing can be optimized.

[0039] The channel number corresponds to the identifier of the physical or virtual communication channel used by each GPU when communicating. Usually, in distributed computing or multi-GPU systems, communication channels are used for data transmission paths or synchronization signal transmission, and different channels can be used for different data streams or operation types. In a multi-GPU parallel computing system, different GPUs may need to exchange data through multiple physical or logical channels. The channel number is a unique identifier to distinguish these channels, which can help the system accurately specify and monitor different communication paths during execution. Channel management is crucial to improving data transmission efficiency, avoiding bottlenecks, and reducing conflicts.

[0040] The execution status information of the currently executed collective communication operation may include three states:

[0041] 1) Not started: indicates that the current communication operation has not been started;

[0042] 2) Executing: indicates that the current communication operation is in progress, which may be the process of data transmission or computing collaboration;

[0043] 3) Execution completed: indicates that the communication operation has been completed and the related data or synchronization operations have been successfully completed.

[0044] The execution status information of collective communication operations is crucial for dynamic scheduling and resource management. In a multi-GPU parallel computing system, monitoring the execution status of collective communication operations can help the system determine which GPUs have completed their tasks, which are waiting for communication results, and which have not yet started processing. Based on this information, the upper-level scheduling system can also make intelligent decisions to avoid resource waste and performance bottlenecks, thereby improving the efficiency of the overall system.

[0045] Step 202: for each GPU having the same channel number, determining a first diagnosis result corresponding to the channel level based on the corresponding executed collective communication operation count and execution state information;

[0046] On the basis of step 201, this step aims to determine, by the execution subject, for each GPU having the same channel number, a first diagnostic result corresponding to the channel level based on the corresponding executed aggregate communication operation count and execution status information.

[0047] For a training task assigned to a computing cluster, the training task is usually divided into multiple communication groups, each of which contains multiple channels, and each GPU may be in different channels at the same time. Therefore, the channel-level fault GPU diagnosis performed in this step is for each GPU with the same channel number, that is, by obtaining the number of executed collective communication operations and the execution status information of the currently executed collective communication operations of each of these GPUs, the first diagnosis result is finally determined based on the comparison of these information.

[0048] Generally, for each GPU in any channel, the GPU with the smallest number of executed aggregate communication operations is most likely to be the faulty GPU, and the probability that other GPUs closely associated with it are faulty GPUs is also relatively high. Similarly, the GPU that does not have the same execution state information as other GPUs is most likely to be the faulty GPU.

[0049] Step 203: Summarize the first diagnostic results corresponding to the channels belonging to the same communication group, and determine the second diagnostic result corresponding to the communication group level;

[0050] On the basis of step 202, this step aims to have the above-mentioned execution subject summarize the first diagnosis results corresponding to each channel belonging to the same communication group, and determine the second diagnosis result corresponding to the communication group level. Specifically, the summarization method can include multiple methods of obtaining the intersection method.

[0051] Step 204: Summarize the second diagnosis results corresponding to the communication groups belonging to the same training task, and determine the target diagnosis result corresponding to the training task level;

[0052] On the basis of step 203, this step aims to have the above-mentioned execution subject summarize the second diagnosis results corresponding to each communication group belonging to the same training task, and determine the target diagnosis result corresponding to the training task level. Specifically, the summarization method can include multiple methods of obtaining the intersection method.

[0053] It should be understood that step 203 and step 204 are intended to determine in a summary manner which are truly faulty GPUs and which are non-faulty GPUs that are erroneously determined to be faulty GPUs due to the influence of truly faulty GPUs, that is, to eliminate wrong answers in a summary manner.

[0054] Step 205: Determine the faulty GPU that affects the normal execution of the current training task based on the target diagnosis result.

[0055] On the basis of step 204, this step aims to determine the faulty GPU that affects the normal execution of the current training task based on the target diagnosis result by the execution subject, for example, directly determining the faulty GPU included in the target diagnosis result as the faulty GPU.

[0056] The method for determining a faulty graphics processing chip in a computing cluster provided by the embodiments of the present disclosure obtains the number of collective communication operations executed, the channel number to which they belong, and the execution status information of the collective communication operation currently being executed for each GPU when executing a collective communication operation, and performs a multi-level and multi-dimensional comprehensive analysis on the obtained information related to the executed collective communication operations of each GPU, thereby finally determining the faulty GPU that causes the current training task to hang, thereby improving the efficiency of troubleshooting faulty nodes in a multi-node computing cluster, minimizing the duration of training tasks hanging due to faults, and ultimately reducing the overall time consumed in executing a single training task.

[0057] Please refer to Figure 3 , Figure 3 A flowchart of a method for diagnosing a faulty GPU at a channel level is provided in an embodiment of the present disclosure, wherein process 300 includes the following steps:

[0058] Step 301: For each GPU having the same channel number, the corresponding executed collective communication operation count and execution status information are obtained;

[0059] Step 302: determine whether the acquired executed aggregate communication operands are completely the same and whether the execution status information is all completed, and select different processing branches to execute according to different situations;

[0060] Step 303: Determine the GPU with the smallest number of executed collective communication operations as the first diagnosis result corresponding to the current channel number at the channel level;

[0061] This step is based on the judgment result of step 302 that the numbers of executed collective communication operations are not exactly the same and regardless of whether the execution status information of each execution is completed, and is intended to allow the above-mentioned execution subject to determine the GPU with the smallest number of executed collective communication operations as the first diagnostic result corresponding to the current channel number at the channel level.

[0062] Step 304: determining that no faults occur as a first diagnosis result corresponding to the current channel number at the channel level;

[0063] This step is based on the judgment result of step 302 that the number of communication operations of each executed set is exactly the same and the execution status information of each execution is completed. It is intended for the above-mentioned execution subject to determine that no fault has occurred as the first diagnostic result corresponding to the current channel number at the channel level, that is, no fault has occurred in each GPU under the current channel number.

[0064] Step 305: Obtain the data volume size set by the user for each GPU having the same channel number;

[0065] This step is based on the judgment result of step 302 that the number of communication operations of each executed set is not exactly the same and the execution status information of each execution is not complete. It is intended for the above-mentioned execution subject to obtain the data size (i.e., Message Size) set by the user for each GPU with the same channel number.

[0066] Step 306: Determine whether the sizes of the acquired data are completely the same. If not, execute step 307; otherwise, execute step 308;

[0067] Step 307: outputting a third fault diagnosis that the user has set an incorrect data volume size for different GPUs in the same channel;

[0068] This step is based on the judgment result of step 306 that the sizes of the acquired data are not exactly the same, and is intended to enable the above-mentioned execution subject to output a third fault diagnosis that the user has set an incorrect data size for different GPUs of the same channel, that is, there is a fault in the upper-level software that sends the training task, and a prompt or notification can be returned to suggest checking whether there is any abnormality in the upper-level software logic.

[0069] Step 308: Determine the actual operation type of the collective communication operation currently executed by each GPU having the same channel number;

[0070] This step is based on the judgment result of step 306 that the sizes of the acquired data are exactly the same, and is intended to determine, by the above-mentioned execution subject, the actual operation type of the collective communication operation currently executed by each GPU with the same channel number.

[0071] Among them, the actual operation types can include two categories, namely, asymmetric collective communication operations represented by Reduce (Chinese can be translated as: aggregation) and Broadcast, and symmetric collective communication operations represented by All Reduce (Chinese can be translated as: full protocol or broadcast protocol), All Gather (Chinese can be translated as: full collection), and Reduce Scatter (Chinese can be translated as: scattered collection or aggregation distribution).

[0072] The characteristics of asymmetric collective communication operations are: all the data blocks to be processed (chunks) start from a certain GPU and are sent unidirectionally in a fixed sending order until the last GPU receives it, which can be called a one-way chain; the characteristics of symmetric collective communication operations are: all GPUs may be the sending source of a certain data block to be processed until it is received by the predecessor GPU of the current GPU, which can be relatively called a two-way chain.

[0073] Step 309: Obtain the number of processed data blocks of each GPU having the same channel number;

[0074] This step aims to obtain the number of processed data blocks of each GPU having the same channel number by the above-mentioned execution subject.

[0075] Step 310: Based on the acquired number of processed data blocks and the actual operation type, determine a first diagnosis result corresponding to the current channel number at the channel level.

[0076] On the basis of step 308 and step 309, this step aims to determine, by the execution subject, the first diagnostic result corresponding to the current channel number at the channel level based on the acquired number of processed data blocks and the actual operation type.

[0077] like Figure 4-1 Two specific cases are shown:

[0078] Case 1: When the actual operation type is specifically a symmetric collective communication operation, the first GPU having the smallest number of processed data blocks and the preceding GPU of the first GPU are jointly determined as the first diagnosis result corresponding to the current channel number at the channel level.

[0079] In symmetric collective communication operations, all GPUs in the chain may be the source of a data block to be processed until it is received by the predecessor GPU of the current GPU. Therefore, when a hang failure occurs, the number of processed data blocks of each GPU (which can be referred to by a counter) will be as follows: Figure 4-2 Features shown:

[0080] That is, the counter size of the faulty GPU must be at the trough in the entire chain. Therefore, the GPU with the smallest counter and its predecessor GPU should be the final set of faulty GPUs in the current channel.

[0081] Case 2: When the actual operation type is specifically an asymmetric collective communication operation, the second GPU that first has the smallest number of processed data blocks in the data flow and the preceding GPU of the second GPU are jointly determined as the first diagnosis result corresponding to the current channel number at the channel level.

[0082] In symmetric collective communication operations, all chunks are sent one-way from a certain GPU in a fixed order until the last GPU receives them. When a hang occurs, the data sent from the preceding GPU of the failed link to the failed GPU cannot be received. Therefore, its counter characteristics are as follows: Figure 4-3 As shown: It decreases monotonically starting from the data sending source GPU and remains unchanged after the faulty GPU. Therefore, the GPU with the smallest counter and its predecessor GPU should be the final faulty GPU set of the current channel.

[0083] This embodiment provides faulty GPU diagnostic solutions for a variety of different situations at the channel level through steps 301-310, wherein the diagnostic solution for each situation contained therein can independently form an independent embodiment with the embodiment provided by process 200, and there is no causal and dependent relationship between the diagnostic solutions in each situation. This embodiment exists only as a preferred embodiment that simultaneously embodies the diagnostic solutions for various situations.

[0084] Based on any of the above examples, see Figure 5 , Figure 5 A flowchart of a method for summarizing and eliminating erroneous diagnosis results by obtaining intersections is provided in an embodiment of the present disclosure, wherein the process 500 includes the following steps:

[0085] Step 501: Obtain an intersection of first diagnosis results corresponding to different channels belonging to the same communication group to obtain a first intersection result;

[0086] Step 502: Determine the GPU recorded in the first intersection result as the second diagnosis result corresponding to the current communication group at the communication group level;

[0087] Step 503: finding the intersection of the second diagnosis results corresponding to different communication groups belonging to the same training task, to obtain a second intersection result;

[0088] Step 504: Determine the GPU recorded in the second intersection result as the target diagnosis result corresponding to the current training task at the training task level;

[0089] Step 505: Determine the GPU recorded in the target diagnosis result as a faulty GPU that affects the normal execution of the current training task.

[0090] That is, this embodiment provides a final determination solution for the faulty GPU by eliminating wrong answers by obtaining the intersection of different channels under the communication group level and different communication groups under the training task level through steps 501 to 505, that is, only the faulty GPU that is the result of the intersection is the real faulty GPU.

[0091] Based on any of the above examples, see Figure 6 , Figure 6 A flowchart of a fault repair method for a faulty GPU provided by an embodiment of the present disclosure, wherein the process 600 includes the following steps:

[0092] Step 601: diagnose the fault type of the faulty GPU and determine the target fault type;

[0093] Step 602: Perform fault repair on the faulty GPU according to a fault repair method corresponding to the target fault type, and obtain a fault repair result;

[0094] All feasible or operable fault repair methods may be configured in advance for different known fault types, such as an automatic repair method, a manual repair method, or a fault repair method of replacing a GPU.

[0095] Step 603: In response to the fault repair result being that the fault has been repaired, continue to execute the current training task.

[0096] This embodiment aims to attempt to repair the faulty GPU located according to the solution provided in the above embodiment, so as to eliminate the hanging problem as soon as possible through the fault repair method and continue to execute the current training task.

[0097] To deepen understanding, this disclosure also provides a specific implementation solution in combination with a specific application scenario. The overall architecture design can be found in Figure 7-1 As shown, Figure 7-1 middle:

[0098] 1. Communication status profiling (data collection, information gathering): When the training task is running normally, the AI ​​communication library (or AI collective communication library, CCL) collects statistics on the collective communication status in real time;

[0099] 2. Fault status output: When the fault detection process finds that the training task is hung, it notifies the training task process to print the collective communication status of all communication groups at the current hang time through Unix socket (Chinese name: Unix domain socket) or other inter-process communication methods;

[0100] 3. Hang node analysis: The training task fault diagnosis controller collects all TraceHang (literally translated as: hang tracking) logs of the task, analyzes the faulty nodes, and outputs them to the user.

[0101] To support the above diagnostic architecture, the following needs to be implemented:

[0102] 1. Collective Communication Library (CCL): responsible for real-time statistics of collective communication status information, and printing the failed collective communication status information when the training task fails:

[0103] 1. When the communication group is initialized, a UM memory (i.e. shared memory) is allocated to each channel of each rank (i.e. GPU with a unique number) of each communication group to record the TraceHang counter. The underlying memory implements video memory and memory mapping through MMAP (Memory Mapped Access Protocol, a memory-oriented access protocol used to map a file or other object into memory) similar logic, and can be accessed by both GPU and CPU at the same time;

[0104] 2. When the application layer submits each collective communication operation, it submits the TraceHang counter as a parameter to the GPU through launchKernel (an important concept in GPU programming, used to start kernel execution on the GPU), and records the message size (i.e. the data size described above), opCount (i.e. the number of executed collective communication operations described above), channel (i.e. the channel number described above) and other information of the collective communication operation;

[0105] 3. When executing the collective communication algorithm on the GPU side, use the TraceHang counter to record the number of chunks processed in the receiving and sending directions of the current block (corresponding channel) when processing the current collective communication operation;

[0106] 4. Mark the status of the current collective communication operation on the GPU side, such as: init (not started), processing (in progress), done (execution completed);

[0107] 5. After the training task hangs, the AI ​​communication library will receive the external Unix socket signal and print the TraceHang counter of all ranks of all communication groups into the log file on the CPU side.

[0108] 2. Training task fault diagnosis controller:

[0109] 1. Monitor the running status of training tasks and find out if training tasks are hung in time;

[0110] 2. When the training task is found to be hung, a unixsocket signal is sent to the training task process of all worker nodes to trigger the communication library log printing;

[0111] 3. After the communication library completes log printing, collect the training task logs of all nodes;

[0112] 4. Perform centralized diagnosis and analysis based on the fault pattern.

[0113] 3. Core fault diagnosis logic:

[0114] Step 1: Aggregate all data of a single task according to the communication group ID. The diagnostic logic of a single communication group is detailed in step 2. After the diagnosis of a single communication group is completed, aggregate the diagnostic results of all communication groups, as detailed in step 6;

[0115] Step 2: For all data of the same communication group, aggregate them according to the channel ID. For the diagnostic logic of a single channel, see step 3. After the diagnosis of a single channel is completed, aggregate the diagnostic results of all channels in a single communication group, see step 5 for details;

[0116] Step 3: For the data of the same channel, the diagnostic process can be found in Figure 3 , where the chain diagnosis process is detailed in step 4. If the diagnosis result is a software fault, then the fault is the root cause of the training task hang failure;

[0117] Step 4: The communication library will perform a channel search process during initialization. Each channel searches for a communication link and submits it to the GPU. All collective communication operations can be roughly divided into two categories: asymmetric collective communication operations and symmetric collective communication operations. The faulty GPU set can be accurately determined based on different types of collective communication operations combined with the count of executed data blocks;

[0118] Step 5: After confirming the fault rank of a single channel, it is necessary to summarize the results of all channels in a single communication group. According to the principle of hang failure, if a certain rank is the root cause of the failure, all faulty channels in the communication group will point to this rank. Therefore, it is necessary to take the intersection of the fault ranks of all faulty channels in the communication group, which is the faulty node of the communication group;

[0119] Step 6: After confirming the failure rank of a single communication group, it is necessary to summarize the results of all communication groups of the training task. Training tasks usually use complex hybrid parallel logic, such as tensor parallelism, pipeline parallelism, data parallelism, etc. Figure 7-2As shown in the figure. According to the hang fault principle, if a GPU is the root cause of the fault, all communication groups containing this GPU will point to this GPU. Therefore, it is necessary to filter the diagnosis results of each communication group according to the above principles, and the result obtained is the faulty GPU node finally diagnosed by TraceHang.

[0120] For ease of understanding, the following will be described in conjunction with specific examples:

[0121] 1. Channel diagnosis

[0122] 1.1 Asymmetric Collective Communication Operations

[0123] The data flow direction is: rank0→rank1→...rank7. The characteristics of counter along the data flow direction are: first monotonically decreasing, then remaining unchanged. Assume that the counter of a single channel is as shown in Table 1, rank3 and the preceding node of rank3 are the fault ranks. That is, the diagnosis result is: rank3, rank4.

[0124] Table 1

[0125] rank id counter 0 391 1 375 2 359 3 358 4 358 5 358 6 358 7 358

[0126] 1.2 Symmetric collective communication operations

[0127] The data flow direction is: rank0→rank1→...rank7→rank0. The counter follows the data flow direction: first monotonically decreasing, then monotonically increasing. Assuming that the counter for a single channel is as shown in Table 2, the trough node and its preceding node are faulty nodes. The trough node is rank1. Therefore, the diagnosed faulty nodes are: rank0, rank1.

[0128] Table 2

[0129] rank id counter 0 929 1 921 2 929 3 937 4 945 5 953 6 961 7 953

[0130] 2. Comm (communication group) diagnosis: multi-channel aggregation

[0131] The principle of aggregating multi-channel diagnosis results is: if rank A is a faulty node, then rank A will definitely be diagnosed as a faulty node in any faulty channel.

[0132] Assume that the diagnosis of multiple channels of a single communication group is summarized in Table 3:

[0133] Table 3

[0134]

[0135] 1) Rank 7 is not a faulty node because it has not been diagnosed as a faulty node in channel 1;

[0136] 2) Rank 8 is a faulty node because rank 8 is diagnosed as a faulty node in all faulty channels (channel 0 and channel 1);

[0137] 3) Rank 9 is not a faulty node because it has not been diagnosed as a faulty node in channel 0;

[0138] 4) In summary: the faulty node diagnosed by this comm is rank 8.

[0139] 3. Task diagnosis: multi-comm aggregation

[0140] The principle of aggregating multi-comm diagnosis results is: if rank A is a faulty node, then rank A will definitely be diagnosed as a faulty node in all faulty comms that contain this node.

[0141] Assume that the diagnosis results of multiple communication groups of a single training task are shown in Table 4 below:

[0142] Table 4

[0143]

[0144] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for determining a faulty graphics processing chip in a computing power cluster. The device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0145] like Figure 8As shown, the device 800 for determining a faulty graphics processing chip in a computing power cluster of this embodiment may include: an information acquisition unit 801, a channel-level diagnostic result determination unit 802, a communication group-level diagnostic result determination unit 803, a training task-level diagnostic result determination unit 804, and a faulty GPU positioning unit 805. Among them, the information acquisition unit 801 is configured to respectively acquire the number of executed collective communication operations, channel number and execution status information of the currently executed collective communication operation corresponding to each graphics processing chip GPU constituting the computing power cluster; the channel-level diagnostic result determination unit 802 is configured to determine the first diagnostic result corresponding to the channel level for each GPU with the same channel number based on the corresponding number of executed collective communication operations and execution status information; the communication group-level diagnostic result determination unit 803 is configured to summarize the first diagnostic results corresponding to each channel belonging to the same communication group, and determine the second diagnostic result corresponding to the communication group level; the training task-level diagnostic result determination unit 804 is configured to summarize the second diagnostic results corresponding to each communication group belonging to the same training task, and determine the target diagnostic result corresponding to the training task level; the faulty GPU positioning unit 805 is configured to determine the faulty GPU that affects the normal execution of the current training task based on the target diagnostic result.

[0146] In this embodiment, in the device for determining a faulty graphics processing chip in a computing power cluster 800, the specific processing of the information acquisition unit 801, the channel-level diagnostic result determination unit 802, the communication group-level diagnostic result determination unit 803, the training task-level diagnostic result determination unit 804, and the faulty GPU positioning unit 805 and the technical effects thereof can be referred to respectively. Figure 2 The relevant descriptions of steps 201 - 205 in the corresponding embodiment are not repeated here.

[0147] In some optional implementations of this embodiment, the channel level diagnosis result determination unit 802 may include:

[0148] The executed operation number and execution status information acquisition subunit is configured to acquire the corresponding executed collective communication operation number and execution status information for each GPU having the same channel number;

[0149] The first diagnosis processing subunit is configured to, in response to the obtained executed collective communication operation numbers being not completely the same, determine the GPU having the smallest executed collective communication operation number as the first diagnosis result corresponding to the current channel number at the channel level.

[0150] In some optional implementations of this embodiment, the channel level diagnosis result determination unit 802 may further include:

[0151] The second diagnosis processing sub-unit is configured to, in response to the obtained executed communication operation numbers of each set being exactly the same and each execution status information being completed, determine that no fault has occurred as a first diagnosis result corresponding to the current channel number at the channel level.

[0152] In some optional implementations of this embodiment, the channel level diagnosis result determination unit 802 may further include:

[0153] The data size acquisition subunit is configured to, in response to the acquired executed aggregate communication operation numbers not being completely the same and the execution status information not being completely completed, acquire the data size set by the user for each GPU having the same channel number;

[0154] The third diagnosis processing subunit is configured to output a third fault diagnosis that the user has set an incorrect data size for different GPUs in the same channel in response to the acquired data sizes not being completely the same.

[0155] In some optional implementations of this embodiment, the channel level diagnosis result determination unit 802 may further include:

[0156] The operation type determination subunit is configured to determine the actual operation type of the collective communication operation currently executed by each GPU having the same channel number in response to the sizes of each acquired data being exactly the same; wherein the actual operation type includes a symmetric operation or an asymmetric operation;

[0157] The processed data block acquisition subunit is configured to acquire the number of processed data blocks of each GPU having the same channel number;

[0158] The fourth diagnosis processing subunit is configured to determine a first diagnosis result corresponding to a current channel number at the channel level based on the acquired number of processed data blocks and the actual operation type.

[0159] In some optional implementations of this embodiment, the fourth diagnosis processing subunit may include:

[0160] The fifth diagnosis module is configured to, in response to the actual operation type being a symmetric operation, jointly determine the first GPU having the smallest number of processed data blocks and the preceding GPU of the first GPU as the first diagnosis result corresponding to the current channel number at the channel level.

[0161] In some optional implementations of this embodiment, the fourth diagnosis processing subunit may include:

[0162] The sixth diagnostic module is configured to, in response to the actual operation type being an asymmetric operation, jointly determine the second GPU that first has the smallest number of processed data blocks in the data flow and the preceding GPU of the second GPU as the first diagnostic result corresponding to the current channel number at the channel level.

[0163] In some optional implementations of this embodiment, the information acquisition unit 801 may be further configured to:

[0164] The number of collective communication operations executed, the channel number, and the execution status information of the collective communication operation currently being executed corresponding to each GPU are obtained from the shared memory allocated in advance to each GPU.

[0165] In some optional implementations of this embodiment, the communication group level diagnosis result determination unit 803 may be further configured to:

[0166] Obtaining an intersection of first diagnosis results corresponding to different channels belonging to the same communication group to obtain a first intersection result;

[0167] The GPU recorded in the first intersection result is determined as a second diagnosis result corresponding to the current communication group at the communication group level.

[0168] In some optional implementations of this embodiment, the training task level diagnosis result determination unit 804 may be further configured to:

[0169] Obtaining the intersection of the second diagnosis results corresponding to different communication groups belonging to the same training task to obtain a second intersection result;

[0170] The GPU recorded in the second intersection result is determined as the target diagnosis result corresponding to the current training task at the training task level.

[0171] In some optional implementations of this embodiment, the faulty GPU locating unit 805 may be further configured to:

[0172] The GPU recorded in the target diagnosis result is determined as a faulty GPU that affects the normal execution of the current training task.

[0173] In some optional implementations of this embodiment, the apparatus 800 for determining a faulty graphics processing chip in a computing power cluster may further include:

[0174] A fault type diagnosis unit is configured to perform fault type diagnosis on the faulty GPU and determine a target fault type;

[0175] A fault repair unit is configured to perform fault repair on the faulty GPU according to a fault repair method corresponding to a target fault type, and obtain a fault repair result;

[0176] The training task continuing execution unit is configured to continue executing the current training task in response to the fault repair result being that the fault has been repaired.

[0177] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The apparatus for determining a faulty graphics processing chip in a computing power cluster provided by this embodiment obtains the number of collective communication operations executed, the channel number to which they belong, and the execution status information of the collective communication operation currently being executed for each GPU in a computing power cluster composed of multiple GPUs, and performs a multi-level and multi-dimensional comprehensive analysis on the information related to the executed collective communication operations of each GPU obtained, thereby finally determining the faulty GPU that causes the current training task to hang, thereby improving the efficiency of troubleshooting faulty nodes in a multi-node computing power cluster, minimizing the duration of training tasks hanging due to faults, and finally reducing the overall time consumed in executing a single training task.

[0178] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the method for determining a faulty graphics processing chip in a computing power cluster described in any of the above embodiments when executing.

[0179] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the method for determining a faulty graphics processing chip in a computing power cluster described in any of the above embodiments when executed.

[0180] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, which, when executed by a processor, can implement the method for determining a faulty graphics processing chip in a computing power cluster described in any of the above embodiments.

[0181] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0182] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0183] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0184] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a method for determining a faulty graphics processing chip within a computing power cluster. For example, in some embodiments, the method for determining a faulty graphics processing chip within a computing power cluster may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method for determining a faulty graphics processing chip within the computing power cluster described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the method for determining a faulty graphics processing chip in a computing power cluster in any other appropriate manner (for example, by means of firmware).

[0185] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0186] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0187] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0188] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0189] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0190] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0191] According to the technical solution of the embodiment of the present disclosure, for a computing power cluster composed of multiple GPUs, by obtaining the number of collective communication operations executed, the channel number to which they belong, and the execution status information of the collective communication operation currently being executed for each GPU when executing the collective communication operation, and performing a multi-level and multi-dimensional comprehensive analysis on the various pieces of information related to the executed collective communication operations of each GPU, the faulty GPU that causes the current training task to hang can be finally determined, thereby improving the efficiency of troubleshooting faulty nodes in a multi-node computing power cluster, minimizing the duration of training task hangs due to faults, and ultimately reducing the overall time consumed in executing a single training task.

[0192] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0193] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for determining a faulty graphics processing chip in a computing cluster, comprising: Respectively obtain the number of executed collective communication operations, the channel number, and the execution status information of the currently executed collective communication operation corresponding to each graphics processing chip GPU constituting the computing power cluster; For each GPU having the same channel number, determining a first diagnostic result corresponding to the channel level based on the corresponding executed collective communication operations and the execution state information; Summarize the first diagnostic results corresponding to the channels belonging to the same communication group, and determine the second diagnostic result corresponding to the communication group level; Summarize the second diagnosis results corresponding to the communication groups belonging to the same training task, and determine the target diagnosis result corresponding to the training task level; A faulty GPU that affects normal execution of the current training task is determined based on the target diagnosis result.

2. The method according to claim 1, wherein: The step of determining, for each GPU having the same channel number, a first diagnostic result corresponding to a channel level based on a corresponding number of executed collective communication operations and execution state information, comprises: For each GPU with the same channel number, the corresponding executed collective communication operation count and execution status information are obtained; In response to the acquired executed collective communication operation numbers being not completely the same, a GPU having a minimum executed collective communication operation number is determined as a first diagnosis result corresponding to a current channel number at the channel level.

3. The method according to claim 2, further comprising: In response to the acquired executed aggregate communication operation numbers being completely identical and the execution status information being all completed, no faults occurring is determined as a first diagnosis result corresponding to the current channel number at the channel level.

4. The method according to claim 2, further comprising: In response to the acquired executed aggregate communication operation numbers not being completely the same and the execution status information not being completely completed, acquiring the data volume size set by the user for each GPU having the same channel number; In response to the acquired data sizes not being completely the same, a third fault diagnosis is outputted that the user has set an incorrect data size for different GPUs in the same channel.

5. The method according to claim 4, further comprising: In response to the sizes of the acquired data amounts being exactly the same, determining the actual operation type of the collective communication operation currently executed by each GPU having the same channel number; wherein the actual operation type includes a symmetric operation or an asymmetric operation; Get the number of processed data blocks of each GPU with the same channel number; Based on the acquired number of processed data blocks and the actual operation type, a first diagnosis result corresponding to the current channel number at the channel level is determined.

6. The method according to claim 5, wherein: The determining, based on the acquired number of processed data blocks and the actual operation type, a first diagnostic result corresponding to the current channel number at the channel level includes: In response to the actual operation type being the symmetric operation, a first GPU having a minimum number of processed data blocks and a preceding GPU of the first GPU are jointly determined as a first diagnosis result corresponding to a current channel number at a channel level.

7. The method according to claim 5, wherein: The determining, based on the acquired number of processed data blocks and the actual operation type, a first diagnostic result corresponding to the current channel number at the channel level includes: In response to the actual operation type being the asymmetric operation, the second GPU that first has the smallest number of processed data blocks in the data flow and a preceding GPU of the second GPU are jointly determined as a first diagnosis result corresponding to the current channel number at the channel level.

8. The method according to claim 1, wherein: The method of respectively obtaining the number of executed collective communication operations, the channel number, and the execution status information of the currently executed collective communication operation corresponding to each graphics processing chip GPU constituting the computing power cluster includes: The number of executed collective communication operations, the channel number, and the execution status information of the currently executed collective communication operation corresponding to each GPU are respectively obtained from the shared memory allocated in advance to each GPU.

9. The method according to any one of claims 1 to 8, wherein: The aggregating the first diagnostic results corresponding to the channels belonging to the same communication group and determining the second diagnostic result corresponding to the communication group level includes: Obtaining an intersection of first diagnosis results corresponding to different channels belonging to the same communication group to obtain a first intersection result; The GPU recorded in the first intersection result is determined as a second diagnosis result corresponding to the current communication group at the communication group level.

10. The method according to claim 9, wherein: The step of aggregating the second diagnosis results corresponding to the communication groups belonging to the same training task and determining the target diagnosis result corresponding to the training task level includes: Obtaining an intersection of the second diagnosis results corresponding to different communication groups belonging to the same training task, to obtain a second intersection result; The GPU recorded in the second intersection result is determined as the target diagnosis result corresponding to the current training task at the training task level.

11. The method according to claim 10, wherein: The determining, based on the target diagnosis result, a faulty GPU that affects normal execution of the current training task includes: The GPU recorded in the target diagnosis result is determined as a faulty GPU that affects the normal execution of the current training task.

12. The method according to claim 11, further comprising: Performing fault type diagnosis on the faulty GPU to determine a target fault type; Performing fault repair on the faulty GPU according to a fault repair method corresponding to the target fault type to obtain a fault repair result; In response to the fault repair result being that the fault has been repaired, the current training task continues to be executed.

13. A device for determining a faulty graphics processing chip in a computing power cluster, comprising: An information acquisition unit is configured to respectively acquire the number of executed collective communication operations, the channel number, and the execution status information of the currently executed collective communication operation corresponding to each graphics processing chip GPU constituting the computing power cluster; a channel-level diagnostic result determination unit configured to determine, for each GPU having the same channel number, a first diagnostic result corresponding to the channel level based on the corresponding executed aggregate communication operation count and execution state information; a communication group level diagnostic result determination unit, configured to aggregate first diagnostic results corresponding to the channels belonging to the same communication group, and determine a second diagnostic result corresponding to the communication group level; A training task level diagnosis result determination unit is configured to aggregate the second diagnosis results corresponding to the communication groups belonging to the same training task, and determine a target diagnosis result corresponding to the training task level; The faulty GPU locating unit is configured to determine the faulty GPU that affects the normal execution of the current training task based on the target diagnosis result.

14. The device according to claim 13, wherein: The channel level diagnostic result determination unit comprises: The executed operation number and execution status information acquisition subunit is configured to acquire the corresponding executed collective communication operation number and execution status information for each GPU having the same channel number; The first diagnosis processing subunit is configured to, in response to the obtained executed collective communication operation numbers being not completely the same, determine the GPU having the smallest executed collective communication operation number as the first diagnosis result corresponding to the current channel number at the channel level.

15. The device according to claim 14, wherein: The channel level diagnostic result determination unit further includes: The second diagnosis processing sub-unit is configured to, in response to the obtained executed communication operation numbers of each set being exactly the same and each execution status information being completed, determine that no fault has occurred as a first diagnosis result corresponding to the current channel number at the channel level.

16. The device according to claim 14, wherein: The channel level diagnostic result determination unit further includes: The data size acquisition subunit is configured to, in response to the acquired executed aggregate communication operation numbers not being completely the same and the execution status information not being completely completed, acquire the data size set by the user for each GPU having the same channel number; The third diagnosis processing subunit is configured to output a third fault diagnosis that the user has set an incorrect data size for different GPUs in the same channel in response to the acquired data sizes not being completely the same.

17. The device according to claim 16, wherein: The channel level diagnostic result determination unit further includes: The operation type determination subunit is configured to determine, in response to the acquired data amounts being exactly the same in size, an actual operation type of the collective communication operation currently executed by each GPU having the same channel number; wherein the actual operation type includes a symmetric operation or an asymmetric operation; The processed data block acquisition subunit is configured to acquire the number of processed data blocks of each GPU having the same channel number; The fourth diagnosis processing subunit is configured to determine a first diagnosis result corresponding to a current channel number at the channel level based on the acquired number of processed data blocks and the actual operation type.

18. The device according to claim 17, wherein: The fourth diagnosis processing subunit comprises: The fifth diagnostic module is configured to, in response to the actual operation type being the symmetric operation, jointly determine the first GPU with the smallest number of processed data blocks and the preceding GPU of the first GPU as the first diagnostic result corresponding to the current channel number at the channel level.

19. The device according to claim 17, wherein: The fourth diagnosis processing subunit comprises: The sixth diagnostic module is configured to, in response to the actual operation type being the asymmetric operation, jointly determine the second GPU that first has the smallest number of processed data blocks in the data flow and the preceding GPU of the second GPU as the first diagnostic result corresponding to the current channel number at the channel level.

20. The device according to claim 13, wherein: The information acquisition unit is further configured to: The number of executed collective communication operations, the channel number, and the execution status information of the currently executed collective communication operation corresponding to each GPU are respectively obtained from the shared memory allocated in advance to each GPU.

21. The device according to any one of claims 13 to 20, wherein: The communication group level diagnosis result determination unit is further configured to: Obtaining an intersection of first diagnosis results corresponding to different channels belonging to the same communication group to obtain a first intersection result; The GPU recorded in the first intersection result is determined as a second diagnosis result corresponding to the current communication group at the communication group level.

22. The device according to claim 21, wherein The training task level diagnosis result determination unit is further configured to: Obtaining an intersection of the second diagnosis results corresponding to different communication groups belonging to the same training task, to obtain a second intersection result; The GPU recorded in the second intersection result is determined as the target diagnosis result corresponding to the current training task at the training task level.

23. The device according to claim 22, wherein: The faulty GPU locating unit is further configured to: The GPU recorded in the target diagnosis result is determined as a faulty GPU that affects the normal execution of the current training task.

24. The apparatus according to claim 23, further comprising: A fault type diagnosis unit is configured to perform fault type diagnosis on the faulty GPU and determine a target fault type; A fault repair unit is configured to perform fault repair on the faulty GPU according to a fault repair method corresponding to the target fault type, and obtain a fault repair result; The training task continuing execution unit is configured to continue executing the current training task in response to the fault repair result being that the fault has been repaired.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for determining a faulty graphics processing chip in a computing power cluster as described in any one of claims 1-12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method for determining a faulty graphics processing chip in a computing power cluster according to any one of claims 1-12.

27. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method for determining a faulty graphics processing chip in a computing power cluster according to any one of claims 1-12.

Images