Fault diagnosis method and apparatus, and electronic device

Abstract

Provided in the embodiments of the present disclosure are a fault diagnosis method and apparatus, and an electronic device. The method is applied to a control node of a cluster, wherein the cluster comprises the control node and a plurality of task nodes, each task node being used for running one or more processes. The method comprises: acquiring collective communication statistics information of each process in a communicator, wherein the communicator is a collection of processes participating in collective communication, and the collective communication statistics information comprises algorithm layer statistics information and / or transport layer statistics information of the collective communication, the algorithm layer statistics information comprising statistics information regarding the processes calling a collective communication interface, and the transport layer statistics information comprising statistics information regarding a transport layer transmitting collective communication data; and on the basis of the collective communication statistics information, determining whether each task node corresponding to the processes fails. Fault diagnosis is realized in a timely manner.

Description

Fault diagnosis methods, devices and electronic equipment

[0001] This disclosure claims priority to Chinese Patent Application No. 202411063666.1, filed with the China Patent Office on August 2, 2024, entitled “Fault Diagnosis Method, Apparatus and Electronic Equipment”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of cloud networking, and more specifically, to fault diagnosis methods, apparatus and electronic equipment. Background Technology

[0003] A large model typically refers to a machine learning model with a large number of parameters and a complex structure. Especially in the field of deep learning, large models are usually trained on a large amount of data to capture complex patterns and relationships in order to achieve high-performance prediction and generation tasks.

[0004] Due to the large size and numerous parameters of the models, large-scale model training, compared to traditional small-scale model training, is characterized by a large cluster size and long training time, requiring a large number of accelerator cards and network cards to participate in the training. The more hardware involved, the greater the probability of single-point failures. It can be said that failures are common and difficult to avoid in large-scale model training. Furthermore, because large-scale model training requires collective communication between processes, a single point of failure will inevitably affect the performance of the entire task or even cause the task to fail. Therefore, timely fault diagnosis is crucial. Summary of the Invention

[0005] This disclosure provides fault diagnosis methods, apparatus, and electronic equipment that enable timely fault diagnosis.

[0006] In a first aspect, this disclosure provides a fault diagnosis method applied to the control node of a cluster, the cluster including the control node and multiple task nodes, each task node being used to run one or more processes; the method includes:

[0007] Obtain the collective communication statistics of each process in the communication sub-sub ...

[0008] Based on the collected communication statistics, it is determined whether the task nodes corresponding to each process are faulty.

[0009] Secondly, this disclosure provides a fault diagnosis method applied to the control node of a large model training cluster, wherein the large model training cluster includes a control node and multiple task nodes, each task node being used to run one or more training processes; the method includes:

[0010] Obtain the collective communication statistics of each training process in the communication sub-sub ...

[0011] Based on the collected communication statistics, it is determined whether the task nodes corresponding to each training process are faulty.

[0012] Thirdly, this disclosure provides a fault diagnosis device applied to the control node of a cluster, the cluster including the control node and multiple task nodes, each task node being used to run one or more processes; the device includes:

[0013] The acquisition module is used to acquire the collective communication statistics of each process in the communication sub-sub ...

[0014] The diagnostic module is used to determine whether the task nodes corresponding to each process are faulty based on the collected communication statistics.

[0015] Fourthly, this disclosure provides an electronic device, including: a memory and a processor;

[0016] The memory is used to store computer programs;

[0017] The processor is configured to execute a computer program stored in the memory, wherein the computer program, when executed, causes the processor to perform the method described in the first aspect or the second aspect.

[0018] Fifthly, this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the method as described in the first or second aspect.

[0019] In a sixth aspect, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in the first or second aspect.

[0020] The fault diagnosis method, apparatus, and electronic device provided in this disclosure utilize the characteristics of aggregated communication between processes in a cluster. It statistically analyzes various data points at the transport and algorithm layers during the aggregated communication process. By using these statistical information, it determines whether anomalies exist at the algorithm and transport layers during the aggregated communication process. Anomalies at the algorithm layer can identify accelerator card failures in task nodes, and anomalies at the transport layer can identify network card failures in task nodes. Therefore, the statistical information from aggregated communication can determine whether a task node is faulty. During cluster operation, fault points can be diagnosed promptly using the aggregated communication statistical information. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 is a schematic flowchart of a fault diagnosis method provided in an embodiment of this disclosure;

[0023] Figure 2 is a schematic diagram of an application scenario of a fault diagnosis method provided in an embodiment of this disclosure;

[0024] Figure 3 is a visualization diagram of aggregated communication statistics provided in an embodiment of this disclosure;

[0025] Figure 4 is a second visualization diagram of aggregated communication statistics provided in an embodiment of this disclosure;

[0026] Figure 5 is a visualization diagram of aggregated communication statistics provided in an embodiment of this disclosure.

[0027] Figure 6 is a visualization diagram of the aggregated communication statistics provided in an embodiment of this disclosure;

[0028] Figure 7 is a visual schematic diagram of the aggregated communication statistics provided in the embodiments of this disclosure;

[0029] Figure 8 is a visualization diagram of aggregated communication statistics provided in an embodiment of this disclosure;

[0030] Figure 9 is a schematic flowchart of a fault diagnosis method provided in an embodiment of this disclosure;

[0031] Figure 10 is a schematic diagram of the structure of a fault diagnosis device provided in an embodiment of this disclosure;

[0032] Figure 11 is a schematic block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0034] The technical terms used in the embodiments of this disclosure will be introduced.

[0035] Group communication: refers to communication between a group of processes, where all processes in the group send and receive data according to certain rules.

[0036] Communicator: An important concept in parallel and distributed computing, a communicator defines a group of processes that can communicate with each other and provides a context in which message passing operations are independent between different communicators. A communicator contains a set of processes, each with a unique identifier within the communicator, usually called its rank.

[0037] Collective Communication Library (CCL): Provides a set of efficient operations for exchanging data between multiple processes or nodes.

[0038] Calibrating Collective Communication over Converged ethernet–Diagnosis (C4D): The process used for fault diagnosis in this embodiment of the disclosure may be referred to as the diagnostic process.

[0039] Large models: These refer to deep neural network models with a large number of parameters (such as billions or tens of billions) and complex structures. After a specialized training process, they are able to process massive amounts of data and complete various complex tasks.

[0040] Accelerator cards are hardware devices specifically designed to accelerate specific computing tasks and are widely used in high-performance computing (HPC), artificial intelligence (AI), machine learning (ML), deep learning (DL), data analysis, and graphics processing. Common examples of accelerator cards include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), neural processing units (NPUs), digital signal processors (DSPs), intelligence processing units (IPUs), and vision processing units (VPUs).

[0041] Training large models typically requires substantial computing resources and distributed computing techniques to complete the training process within a reasonable timeframe. Aggregate communication is particularly important in large model training because it enables efficient data exchange and synchronization across multiple nodes. A large model training cluster usually consists of multiple task nodes, each running one or more processes. Each process corresponds to an accelerator card and a network interface card (NIC) on the task node. Aggregate communication between processes—that is, the corresponding accelerator cards sending and receiving data through their respective NICs—is crucial. The existence of aggregate communication in large model training creates a single point of failure within the cluster; a failure of any accelerator card or NIC will inevitably impact the performance of the entire task or even cause it to fail. Therefore, timely diagnosis of single points of failure is essential for timely troubleshooting.

[0042] Therefore, this disclosure proposes a fault diagnosis method that utilizes the characteristics of aggregated communication between processes in a cluster. It statistically analyzes various data points at the transport and algorithm layers during the aggregated communication process. By using these statistical information, it determines whether any anomalies exist at the transport and algorithm layers during the aggregated communication process. Specifically, if an anomaly exists at the transport layer (i.e., at the communication layer), a network card anomaly can be identified; if an anomaly exists at the algorithm layer (i.e., at a non-communication layer), an accelerator card anomaly can be determined. By utilizing the statistical information of each process, the fault location can be accurately identified.

[0043] Figure 1 is a flowchart illustrating a fault diagnosis method provided in an embodiment of this disclosure. This method is applied to the control node of a cluster, which includes a control node and multiple task nodes. Each task node runs one or more processes, and each process corresponds to an accelerator card and a network card on the task node. The cluster in this embodiment can be a large model training cluster or other computing clusters based on cluster communication; this embodiment is not limited in this regard. As shown in Figure 1, the method includes:

[0044] S101. Obtain the collective communication statistics of each process in the communicator, wherein the communicator is a set of processes participating in collective communication, and the collective communication statistics include algorithm layer statistics and / or transport layer statistics of collective communication.

[0045] The algorithm layer statistics include statistics on process calls to the collection communication interface, and the transport layer statistics include statistics on the transport layer's transmission of collection communication data.

[0046] A communication sub-sub-component comprising a group of processes participating in collective communication can be processes on one or more task nodes within a cluster. For example, a cluster might have 100 task nodes, each running two processes corresponding to the node's two accelerator cards and two network cards. Taking a communication sub-sub-component composed of processes from 10 task nodes as an example, this means the sub-sub-component contains 20 processes from these 10 task nodes. Each process has a unique identifier within the communication sub-sub-component, for example, from rank0 to rank19. These 20 processes perform collective communication by calling the collective communication interface in the collective communication library to achieve data transfer. Commonly used collection communication interfaces include: Broadcast, which sends a message from one process within a communication subprocess to all other processes within the subprocess; Gather, which collects data from all processes within a communication subprocess to a root process; Scatter, which distributes data from the root process within a communication subprocess to all other processes within the subprocess; Allgather, which collects data from all processes within a communication subprocess and distributes it to all processes; Alltoall, where each process within a communication subprocess sends data to all other processes and receives data from all other processes; Reduce, which performs reduction operations (such as summation, maximum value calculation, etc.) on the data from all processes within a communication subprocess and sends the result to the root process; and Allreduce, which performs reduction operations on the data from all processes within a communication subprocess and distributes the result to all processes.

[0047] Taking Allreduce as an example, assuming processes in a communication subsystem transmit data through the Allreduce interface, if the network card corresponding to the process is faulty, it may manifest as excessively long Remote Direct Memory Access (RDMA) transmission times or incomplete RDMA requests. If the accelerator card corresponding to the process is faulty, such as slow computation speed or freezing, it may manifest as the process being unable to enter the next collective communication or entering the collective communication later than other processes. In other words, by statistically analyzing various statistics generated by processes calling the collective communication interface, such as communication time, data transmission size, number of collective communication requests, and number of successful or completed requests, we can determine whether there are any anomalies in the collective communication, and thus whether the accelerator card or network card involved in the collective communication is faulty.

[0048] Data transmission between processes via the collective communication interface includes algorithm-level calls and data transmission through the network interface card (NIC). This encompasses both algorithm-level interface calls and transport-level data transmission. Therefore, collective communication statistics can include both algorithm-level and transport-level statistics. For example, algorithm-level statistics might include communication time, data size, number of requests, and the number of successful or completed requests for the collective communication interface. Transport-level statistics, on the other hand, might include transport-level requests, such as RDMA requests, including communication time, data size, number of requests, and the number of successful or completed requests.

[0049] The collective communication statistics of each process can be sent from the task node to the control node. For example, Figure 2 illustrates an application scenario of a fault diagnosis method provided in this embodiment. As shown in Figure 2, the cluster includes a control node and multiple task nodes. The processes on the task nodes implement collective communication through a collective communication library. In this embodiment, diagnostic processes can be deployed on the control node and task nodes respectively. The diagnostic process on the control node is the master diagnostic process. When the processes on the task nodes that perform collective communication call the collective communication interface to transmit data, the diagnostic process can record the aforementioned collective communication statistics and report the statistics to the master diagnostic process on the control node.

[0050] S102. Determine whether there is a fault in the task node corresponding to each process based on the collective communication statistics.

[0051] The algorithm-level statistics of aggregate communication reflect the statistics of process calls to the aggregate communication interface. This information can indicate whether there are anomalies at the non-communication level. For example, in an aggregate communication session, if one process calls the interface significantly later than the others, it might be due to excessive time spent in the initial computation phase of that process—a non-communication level anomaly, possibly caused by an accelerator card malfunction. The transport-level statistics of aggregate communication reflect the statistics of data transmission through the network card after a process calls the aggregate communication interface. This information can indicate whether there are anomalies at the communication level. For example, in an aggregate communication session, if an RDMA request sent by one process to another process never completes, it indicates a communication level anomaly, possibly caused by a hang in the link between the network cards of the two processes. Algorithm-level statistics can determine whether there are anomalies at the non-communication level, i.e., whether the accelerator card is faulty; transport-level statistics can determine whether there are anomalies at the communication level, i.e., whether the network card is faulty. In other words, algorithm-level statistics determine whether the accelerator card for each process is faulty, and / or transport-level statistics determine whether the network card for each process is faulty.

[0052] Because it's possible to statistically analyze the communication between each process, it's possible to accurately determine which process or which two processes are experiencing an anomaly, thus accurately identifying the faulty accelerator card or network card. The fault types of the accelerator card or network card can include the card / card being frozen, the link between network cards being frozen, slow computation on the accelerator card, or network card congestion.

[0053] The fault diagnosis method provided in this disclosure utilizes the characteristics of aggregated communication between processes in a cluster. It statistically analyzes various data points at the transport and algorithm layers during the aggregated communication process. By using these statistics, it determines whether anomalies exist at the algorithm and transport layers during the aggregated communication process. Anomalies at the algorithm layer can identify accelerator card failures in task nodes, and anomalies at the transport layer can identify network card failures. Therefore, aggregated communication statistics can determine whether a task node is faulty. During cluster operation, real-time aggregated communication statistics enable timely diagnosis of fault points.

[0054] The following section explains how to determine faults based on collective communication statistics, taking into account the specific content contained therein.

[0055] First, it should be noted that aggregate communication statistics can be collected separately for different aggregate communication interfaces, such as statistics for Allreduce. Furthermore, statistics can be categorized according to different algorithms, data types, and element counts under the same aggregate communication interface, and statistics can be collected separately for each category. For example, data types can be float16, float32, uint32, etc., and the element count refers to the amount of data of a certain data type sent. At the transport layer, in addition to the network card corresponding to the accelerator card, statistics can also be collected separately for RDMA queue pair (QP) connections. In the fault diagnosis method of this embodiment, when comparing aggregate communication statistics between processes, the statistics are compared within the same category, for example, comparing the statistics for each process calling the Allreduce interface, sending float16 data, and having 8 elements.

[0056] In one scenario, transport layer statistics include the number of transport layer requests and the number of request completions between processes. Taking a communication subsystem comprising eight processes (rank 0 to rank 7) as an example, the transport layer statistics are based on the communication between these eight processes using a collective communication interface. rank 0 to rank 7 use the collective communication interface for collective communication. At the transport layer, the network cards corresponding to rank 0 to rank 7 transmit data via RDMA. The number of transport layer requests and the number of request completions between processes are the RDMA request count and the RDMA request completion count. Referring to Figure 3, which shows an example of rank 0 to rank 7 sending data cyclically through the collective communication interface, the transport layer request counts and request completion counts for rank 0 to rank 1, rank 1 to rank 2, and up to rank 7 to rank 0 are statistically analyzed, showing the request count and request completion count for each connection pair.

[0057] In this scenario, if the number of transport layer requests from the first process to the second process remains unchanged and the time between the number of requests completed exceeds a first preset time, it is determined that the network card corresponding to the first process and / or the second process is faulty, and the type of fault is link hang.

[0058] If the transport layer links between the network cards corresponding to the two processes are normal, the transport layer request will complete after a period of time, i.e., the request is complete. In this case, one transport layer request corresponds to one request completion, and the next transport layer request can then proceed. However, if the link is dead, the transport layer request will not complete after transmission, i.e., no request is completed. In this case, the number of completed transport layer requests will be less than the number of requests, and the next transport layer request cannot proceed. If the number of transport layer requests from the first process to the second process remains unchanged but is inconsistent with the number of completed requests, it indicates that there are still uncompleted transport layer requests from the first process to the second process, and the next transport layer request cannot proceed. If this state continues for more than a first preset time, it indicates that the transport layer requests from the first process to the second process cannot be completed in a long time. At this point, it can be determined that the link between the network cards corresponding to the first process and the second process is dead, therefore, the transport layer requests from the first process to the second process cannot be completed. As shown in Figure 3, after the 100th transport layer request, the 100th transport layer request from rank4 to rank5 is still not completed after more than 10 seconds. At this point, it is determined that the link between the network cards from rank4 to rank5 is dead.

[0059] In one scenario, transport layer statistics include the time taken for transport layer requests between processes and the size of the transmitted data. The time taken for a transport layer request can be determined by recording the start and end times of the request. Based on the time taken and the size of the transmitted data, the throughput of transport layer requests between processes is determined. Based on the throughput, it is determined whether the network interface card (NIC) corresponding to each process is faulty. Throughput is the amount of data transmitted per unit time; therefore, the throughput of transport layer requests between processes can be obtained by dividing the transmitted data size by the time taken for the transport layer request. A high throughput indicates low communication overhead, while a low throughput indicates high communication overhead. Excessive communication overhead indicates a possible anomaly at the communication layer, i.e., a possible NIC failure.

[0060] Figures 4, 5, and 6 illustrate the communication overhead between processes. Taking a communication subprocess containing eight processes (ranks 0 to 7) as an example, grayscale is used in Figures 4, 5, and 6 to represent the level of communication overhead, with higher grayscale indicating higher overhead. Furthermore, Figures 4, 5, and 6 assume that all ranks transmit data to each other.

[0061] In one scenario, if the throughput of the transport layer request from the third process to the fourth process is lower than the throughput between other processes, and the difference between the throughput of the transport layer request from the third process to the fourth process and the throughput between other processes is greater than a first threshold, then it is determined that the network interface cards (NICs) corresponding to the third and fourth processes are faulty, and the fault type is link congestion. If a certain network link is congested while other network links are normal, it will manifest as the throughput of that network link being lower than that of other network links. Therefore, by comparing the throughput of the transport layer requests between each process, if the difference between the throughput of the transport layer request from the third process to the fourth process and the throughput between other processes is greater than the first threshold, it indicates that the throughput of the transport layer request from the third process to the fourth process is too low, i.e., the communication overhead is too high. In this case, it is determined that the link between the NICs corresponding to the third and fourth processes is congested, which may be due to a fault in the NIC corresponding to the third process and / or the NIC corresponding to the fourth process. Figure 4 illustrates that the communication overhead of the transport layer request from rank 3 to rank 4 is higher than that of the transport layer request between other ranks, which means that the link between the network cards of rank 3 and rank 4 is congested, and the network cards of rank 3 and / or rank 4 are faulty.

[0062] In one scenario, if the throughput of transport layer requests from the fifth process to any other process is lower than the throughput between other processes, and the difference between the throughput of the fifth process to any other process and the throughput between other processes is greater than a second threshold, then the network interface card (NIC) corresponding to the fifth process is determined to be faulty, and the fault type is transmit-direction congestion. If transmit-direction congestion occurs in a NIC, it will manifest as excessively high data communication overhead for that NIC to send data to all other NICs, i.e., excessively low throughput. Therefore, if it is found that the throughput of transport layer requests from the first process to any other process is lower than the throughput between other processes, and the difference is greater than the second threshold, then it is determined that the data transmission throughput of the NIC corresponding to the fifth process is too low, the transmit-direction congestion of the NIC corresponding to the fifth process, and the NIC corresponding to the fifth process is faulty. Figure 5 illustrates that the communication overhead of transport layer requests from rank3 to all other ranks is higher than the communication overhead of transport layer requests between other ranks, meaning that the transmit-direction congestion of the NIC for rank3 indicates that the NIC for rank3 is faulty.

[0063] In one scenario, if the throughput of transport layer requests from any process to the sixth process is lower than the throughput between other processes, and the difference between the throughput of transport layer requests from any process to the sixth process and the throughput between other processes is greater than a third threshold, then the network interface card (NIC) corresponding to the sixth process is determined to be faulty, and the fault type is receive-direction congestion. If the receive-direction of a NIC is congested, it will manifest as excessively high data communication overhead from all other NICs to that NIC, i.e., excessively low throughput. Therefore, if it is found that the throughput of transport layer requests from any process to the sixth process is lower than the throughput between other processes, and the difference is greater than the third threshold, then it is determined that the data reception throughput of the NIC corresponding to the sixth process is too low, the receive-direction of the NIC corresponding to the sixth process is congested, and the NIC corresponding to the sixth process is faulty. Figure 6 illustrates that the communication overhead of transport layer requests from all other ranks to rank 3 is higher than the communication overhead of transport layer requests between other ranks, meaning that the receive-direction of the NIC at rank 3 is congested, and the NIC at rank 3 is faulty.

[0064] In one scenario, algorithm-level statistics include the number of times a process calls the collective communication interface and the number of successful calls. Taking a communication sub-process comprising eight processes (ranks 0 to 7) as an example, processes rank 0 to 7 call the collective communication interface for collective communication, such as calling the Allreduce interface. The number of interface calls and the number of successful calls for each process are recorded, for example, the number of Allreduce interface calls and the number of successful Allreduce interface calls. If the seventh process has the same number of interface calls and the same number of successful calls, but its number of interface calls is less than that of other processes, and the duration for which the seventh process's number of interface calls remains less than that of other processes exceeds a second preset time, then the accelerator card corresponding to the seventh process is determined to be stuck. In this context, if the number of interface calls and the number of successful interface calls for the seventh process are the same, it indicates that all currently generated interface calls have been completed. However, if the number of interface calls for the seventh process is less than the number of interface calls for other processes, it means that while other processes have entered the next round of aggregate communication, the seventh process has not yet entered the next round. If this period exceeds the second preset time, it indicates that the seventh process has been unable to enter the next round of aggregate communication for an extended period. In this case, it is determined that the accelerator card corresponding to the seventh process is stuck, preventing it from entering the next round of calls. Referring to Figure 7, which illustrates the number of calls and successful calls to the Allreduce interface from rank0 to rank7, it can be seen that while other ranks are entering their 12th call, rank4 has not generated its 12th call for more than 10 seconds, thus determining that the accelerator card for rank4 is stuck.

[0065] In one scenario, algorithm-level statistics include the start time of the interface call for the collective communication. Taking a communication sub-process comprising eight processes (ranks 0 to 7) as an example, processes rank 0 to 7 call the collective communication interface, such as the Allreduce interface. Each process calling the Allreduce interface can record both the start and end times. If, in multiple collective communication calls, the start time of the interface call for the eighth process is later than the start times of the interface calls for other processes, and the difference between the start time of the eighth process's interface call and the start times of the interface calls for other processes exceeds the fourth threshold, and the frequency or proportion of this target condition exceeds the fifth threshold, then the accelerator card corresponding to the eighth process is determined to be slow. The interface call start time of the eighth process is later than that of other processes, indicating that the accelerator card corresponding to the eighth process processes the computing tasks slower than the accelerator cards corresponding to other processes. This results in the interface call start time of the eighth process being later than that of other processes. If the difference between the two is greater than the fourth threshold, it indicates that the accelerator card corresponding to the eighth process is significantly slower than the accelerator cards corresponding to other processes. Furthermore, if this situation occurs too frequently in multiple aggregate communications, exceeding the fifth threshold in frequency or proportion, it means that the accelerator card corresponding to the eighth process consistently fails to complete the computing tasks in a timely manner, leading to a late interface call start time. Therefore, it is determined that the accelerator card corresponding to the eighth process is slow. Referring to Figure 8, which illustrates the non-communication time and communication time for ranks 0 to 7, the non-communication time is, for example, the computing time, and the communication time is the communication time for calling the Allreduce interface. It can be seen that rank 1 transitions from the non-communication time to the communication time, meaning that the interface call start time of the Allreduce interface is always later than that of other ranks. Therefore, it is determined that the accelerator card of rank 1 is slow.

[0066] The above embodiments illustrate how to determine faults by referring to the specific content included in the aggregated communication statistics. It is understood that in applications, one or more of the aforementioned aggregated communication statistics can be obtained as needed to diagnose faults in the accelerator card or network card. In the case of diagnosing a faulty network card or accelerator card, in this embodiment of the disclosure, the fault information can also be sent to the cluster scheduler, enabling the scheduler to isolate the fault point in a timely manner, such as taking the fault point offline and scheduling other task nodes to complete the task execution.

[0067] The fault diagnosis method provided in this disclosure can be applied to any computing cluster based on cluster communication. Taking a large model training cluster as an example, Figure 9 is a flowchart illustrating the fault diagnosis method provided in this disclosure. The method is applied to the control node of a large model training cluster, which includes a control node and multiple task nodes. Each task node is used to run one or more training processes, and each training process corresponds to an accelerator card and a network card on the task node. The method includes:

[0068] S901. Obtain the collective communication statistics of each training process in the communication sub-process, wherein the communication sub-process is a set of training processes participating in collective communication, and the collective communication statistics include the algorithm layer statistics and / or transport layer statistics of collective communication.

[0069] Among them, the algorithm layer statistics include statistics on process calls to the collection communication interface, and the transport layer statistics include statistics on the transport layer transmission of collection communication data;

[0070] S902. Determine whether there is a fault in the task node corresponding to each training process based on the statistical information of the set communication.

[0071] The implementation principle and technical effects of this embodiment are similar to those of the aforementioned embodiments. In scenarios where the cluster is a large model training cluster, aggregated communication is particularly important. Task nodes in a large model training cluster can exchange and synchronize training data and parameters through aggregated communication. Using the method in this embodiment, faulty network cards or accelerator cards in the large model training cluster can be identified, allowing for timely elimination of these faults. For example, if a network card or accelerator card is down, the training task at the faulty point can be stopped, and other task nodes can be scheduled to complete the training. It is understood that in large model training scenarios, training time is usually long. Therefore, in cases of network card congestion or slow accelerator cards, the impact of the fault is a slow training speed, but the training task can still proceed. In this situation, if the training task is nearing completion, scheduling other task nodes may take even longer, so the fault can be left unaddressed.

[0072] Figure 10 is a schematic diagram of a fault diagnosis device provided in an embodiment of this disclosure. As shown in Figure 10, the fault diagnosis device 1000 includes:

[0073] The acquisition module 1001 is used to acquire the collective communication statistics of each process in the communication sub-sub ...

[0074] The diagnostic module 1002 is used to determine whether there is a fault in the task node corresponding to each process based on the collective communication statistics.

[0075] In one implementation, each process corresponds to an accelerator card and a network card on the task node;

[0076] Determining whether there are faults in the task nodes corresponding to each process based on aggregated communication statistics includes:

[0077] The algorithm layer uses statistical information to determine whether the accelerator card corresponding to each process is faulty, and / or uses statistical information from the transport layer to determine whether the network card corresponding to each process is faulty.

[0078] In one implementation, the transport layer statistics include: the number of transport layer requests between processes and the number of requests completed.

[0079] Diagnostic module 1002 is used for:

[0080] If the number of transport layer requests from the first process to the second process remains unchanged and the time between this number and the number of completed requests exceeds a first preset time, then it is determined that the network interface card corresponding to the first process and / or the second process is faulty, and the fault type is a link hang.

[0081] In one implementation, transport layer statistics include: the time taken for transport layer requests between processes and the size of the transmitted data;

[0082] Diagnostic module 1002 is used for:

[0083] The throughput of transport layer requests between processes is determined based on the time taken and the amount of data transmitted.

[0084] Based on the throughput of transport layer requests between processes, determine whether the network interface card corresponding to each process is faulty.

[0085] In one implementation, the diagnostic module 1002 is used for:

[0086] If the throughput of the transport layer request from the third process to the fourth process is lower than the throughput between other processes, and the difference between the throughput of the transport layer request from the third process to the fourth process and the throughput between other processes is greater than the first threshold, then the link between the network cards corresponding to the third process and the fourth process is determined to be congested.

[0087] In one implementation, the diagnostic module 1002 is used for:

[0088] If the throughput of the transport layer request from the fifth process to any other process is lower than the throughput between other processes, and the difference between the throughput of the transport layer request from the fifth process to any other process and the throughput between other processes is greater than the second threshold, then it is determined that the network card corresponding to the fifth process is faulty, and the fault type is transmission direction congestion.

[0089] In one implementation, the diagnostic module 1002 is used for:

[0090] If the throughput of the transport layer request from any process to the sixth process is lower than the throughput between other processes, and the difference between the throughput of the transport layer request from any process to the sixth process and the throughput between other processes is greater than the third threshold, then it is determined that the network card corresponding to the sixth process is faulty, and the fault type is receive-direction congestion.

[0091] In one implementation, the algorithm layer statistics include: the number of times the process calls the set communication interface and the number of successful interface calls;

[0092] Diagnostic module 1002 is used for:

[0093] If the number of interface calls and the number of successful interface calls of the seventh process are the same, but the number of interface calls of the seventh process is less than the number of interface calls of other processes, and the duration for which the number of interface calls of the seventh process remains less than the number of interface calls of other processes exceeds the second preset time, then it is determined that the acceleration card corresponding to the seventh process is dead.

[0094] In one implementation, the algorithm layer statistics include: the start time of the interface call for set communication;

[0095] Diagnostic module 1002 is used for:

[0096] If, in multiple aggregated communications, the interface call start time of the eighth process is later than the interface call start time of other processes, and the difference between the interface call start time of the eighth process and the interface call start time of other processes is greater than the target situation of the fourth threshold, and the number or proportion of the target situation exceeds the fifth threshold, then it is determined that the accelerator card corresponding to the eighth process is slow.

[0097] The fault diagnosis device of this disclosure can be used to implement the fault diagnosis method of any of the foregoing embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0098] Figure 11 is a schematic block diagram of an electronic device provided in an embodiment of this disclosure. As shown in Figure 11, the electronic device 1100 may include at least one processor 1101 for implementing the fault diagnosis method provided in an embodiment of this disclosure.

[0099] Optionally, the electronic device 1100 further includes at least one memory 1102 for storing program instructions and / or data. The memory 1102 is coupled to the processor 1101. The coupling in this embodiment is an indirect coupling or communication connection between devices, units, or modules, and can be electrical, mechanical, or other forms, for information exchange between devices, units, or modules. The processor 1101 may operate in conjunction with the memory 1102. The processor 1101 may execute program instructions stored in the memory 1102. At least one of the at least one memory may be included in the processor.

[0100] Optionally, the electronic device 1100 further includes a communication interface 1103 for communicating with other devices via a transmission medium, thereby enabling the electronic device 1100 to communicate with other devices. The communication interface 1103 may be, for example, a transceiver, interface, bus, circuit, or a device capable of transmitting and receiving functions. The processor 1101 can utilize the communication interface 1103 to transmit and receive data and / or information, and to implement the methods provided in the embodiments of this disclosure. For details, please refer to the detailed descriptions in the preceding embodiments; further elaboration is not required here.

[0101] This embodiment of the present disclosure does not limit the specific connection medium between the processor 1101, memory 1102, and communication interface 1103. In FIG11, the processor 1101, memory 1102, and communication interface 1103 are connected via a bus 1104. The bus 1104 is represented by a thick line in FIG11. The connection methods between other components are only illustrative and not intended to be limiting. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in FIG11, but this does not indicate that there is only one bus or one type of bus.

[0102] It should be understood that the processor in the embodiments of this disclosure can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method embodiments can be completed by the integrated logic circuitry in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0103] It should also be understood that the memory in the embodiments of this disclosure may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0104] This disclosure also provides a hybrid cloud cluster, including a first cluster and a second cluster, wherein the first cluster is a cloud-based cluster and the second cluster is an offline cluster, or the first cluster is an offline cluster and the second cluster is a cloud-based cluster; the first cluster is used to execute the method in any of the foregoing embodiments; the second cluster is used to execute the method in any of the foregoing embodiments.

[0105] This disclosure also provides a computer-readable storage medium storing a computer program (also referred to as code or instructions). When the computer program is run by a processor, it causes the computer to perform the methods as described in any of the foregoing embodiments.

[0106] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the methods in any of the foregoing embodiments.

[0107] As used in this disclosure, the terms “unit”, “module”, etc., can be used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution.

[0108] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure. In the several embodiments provided in this disclosure, it should be understood that the disclosed apparatus, devices, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0109] The unit described as a separate component may or may not be physically separate. The component shown as a unit may or may not be a physical unit; that is, it may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0110] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0111] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. This computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this disclosure is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, magnetic tape), optical media (e.g., digital video disc (DVD)), or semiconductor media (e.g., solid-state disk (SSD)).

[0112] If this function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0113] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.

[0114] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A failure diagnosis method, wherein, The application is applied to a control node of a cluster, the cluster comprising a control node and a plurality of task nodes, each of the task nodes being configured to run one or more processes; the method comprising: obtaining collection communication statistical information of each process in a communication sub, wherein the communication sub is a collection of processes participating in collection communication, the collection communication statistical information comprising algorithm layer statistical information and / or transport layer statistical information of the collection communication, the algorithm layer statistical information comprising statistical information of the processes calling a collection communication interface, and the transport layer statistical information comprising statistical information of the transport layer transmitting collection communication data; judging whether a task node corresponding to each process is faulty based on the collection communication statistical information.

2. The method of claim 1, wherein, Each process corresponds to an acceleration card and a network card on the task node. The judgment of whether the task node corresponding to each process is faulty based on the collection communication statistical information comprises: judging whether an acceleration card corresponding to each process is faulty based on the algorithm layer statistical information, and / or judging whether a network card corresponding to each process is faulty based on the transport layer statistical information.

3. The method of claim 2, wherein, The transport layer statistical information comprises the number of transport layer requests and the number of request completions between processes. The judgment of whether the network card corresponding to each process is faulty based on the transport layer statistical information comprises: if the number of transport layer requests between a first process and a second process remains unchanged and is inconsistent with the number of request completions for a time exceeding a first preset time, it is determined that the network card corresponding to the first process and / or the second process is faulty, and the type of the fault is a link hang-up.

4. The method of claim 2, wherein, The transport layer statistical information comprises the time consumption and the transmission data size of transport layer requests between processes. The judgment of whether the network card corresponding to each process is faulty based on the transport layer statistical information comprises: determining the throughput rate of transport layer requests between the processes based on the time consumption and the transmission data size of transport layer requests between the processes; and determining whether the network card corresponding to each process is faulty based on the throughput rate of transport layer requests between the processes.

5. The method of claim 4, wherein, The determination of whether the network card corresponding to each process is faulty based on the throughput rate of transport layer requests between the processes comprises: if the throughput rate of transport layer requests between a third process and a fourth process is lower than the throughput rate between other processes, and the difference between the throughput rate of transport layer requests between the third process and the fourth process and the throughput rate between other processes is greater than a first threshold value, it is determined that the network card corresponding to the third process and / or the fourth process is faulty, and the type of the fault is link congestion.

6. The method of claim 4, wherein, The determination of whether the network card corresponding to each process is faulty based on the throughput rate of transport layer requests between the processes comprises: if the throughput rate of transport layer requests between a fifth process and any process is lower than the throughput rate between other processes, and the difference between the throughput rate of transport layer requests between the fifth process and the any process and the throughput rate between other processes is greater than a second threshold value, it is determined that the network card corresponding to the fifth process is faulty, and the type of the fault is sending direction congestion.

7. The method of claim 4, wherein, The method comprises: If the throughput rate of the transport layer request from any process to the sixth process is lower than the throughput rate between other processes, and the difference between the throughput rate of the transport layer request from the any process to the sixth process and the throughput rate between other processes is greater than a third threshold value, it is determined that the network card corresponding to the sixth process has a fault, and the type of the fault is congestion in the receiving direction.

8. The method of any one of claims 2-7, wherein, The algorithm layer statistical information comprises: the number of interface calls and the number of successful interface calls of the process calling the collective communication interface; The method comprises: If the number of interface calls and the number of successful interface calls of the seventh process are the same, but the number of interface calls of the seventh process is less than the number of interface calls of other processes, and the duration for which the number of interface calls of the seventh process remains less than the number of interface calls of other processes exceeds a second preset time, it is determined that the accelerator card corresponding to the seventh process is dead.

9. The method of any one of claims 2-7, wherein, The algorithm layer statistical information comprises: the start time of the interface call of the collective communication; The method comprises: If, in a plurality of collective communications, the start time of the interface call of the eighth process is later than the start time of the interface call of other processes, and the difference between the start time of the interface call of the eighth process and the start time of the interface call of other processes is greater than a fourth threshold value, and the number of times or the proportion of the target condition exceeds a fifth threshold value, it is determined that the accelerator card corresponding to the eighth process is slow.

10. A failure diagnosis method in which, The method comprises: Collecting collective communication statistical information of each training process in a communication sub-group, wherein the communication sub-group is a set of training processes participating in collective communication, the collective communication statistical information comprises algorithm layer statistical information and / or transport layer statistical information of the collective communication, the algorithm layer statistical information comprises statistical information of the process calling the collective communication interface, and the transport layer statistical information comprises statistical information of the transport layer transmitting collective communication data; Determining whether a task node corresponding to each training process has a fault based on the collective communication statistical information.

11. A failure diagnosing apparatus, wherein, The method comprises: A collecting module is configured to collect collective communication statistical information of each process in a communication sub-group, wherein the communication sub-group is a set of processes participating in collective communication, the collective communication statistical information comprises algorithm layer statistical information and / or transport layer statistical information of the collective communication, the algorithm layer statistical information comprises statistical information of the process calling the collective communication interface, and the transport layer statistical information comprises statistical information of the transport layer transmitting collective communication data. A diagnosis module is configured to determine whether a task node corresponding to each of the processes has a fault based on the set of communication statistical information.

12. An electronic device, comprising: The method comprises: a memory and a processor; the memory is configured to store a computer program; the processor is configured to execute the computer program stored in the memory, and the computer program, when executed, causes the processor to perform the method in any one of claims 1-10.

13. A computer readable storage medium, wherein, The computer readable storage medium stores a computer program, and the computer program, when executed by a processor, causes the processor to perform the method in any one of claims 1-10.

14. A computer program product, wherein, The computer program, when executed by a processor, implements the method in any one of claims 1-10.

Images