A model training fault diagnosis method and system for collective communication

Abstract

The present invention discloses a model training fault diagnosis method and system for collective communication, which belongs to the field of artificial intelligence and distributed computing. The method comprises: implanting a trigger count to count the number of triggers of the collective communication primitive in the torch_plugin plug-in; adding completion information of processing collective tasks to the communication library, including the counts of various collective communication primitives and the amount of data sent and received, and registering a query interface with the torch_plugin; finally, pre-starting a separate fault diagnosis and analysis thread in different RANK processes of model training, collecting the task counts of the collective communication primitives issued by the upper layer training on different nodes and the task counts actually completed by the collective communication layer, and the amount of data sent and received by the collective communication layer according to the set sampling interval, and analyzing and diagnosing the fault information when the model training is abnormal. The present invention is suitable for large-scale distributed training scenarios and can improve the troubleshooting efficiency of model training tasks.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and distributed computing, and in particular to a model training fault diagnosis method and system for collective communication. Background Art

[0002] Large-scale model training is a key research direction in the field of artificial intelligence, involving the construction and optimization of neural network models with massive parameter counts. With the rapid development of AI technology, the scale and complexity of deep learning models have continued to increase. In recent years, from models initially consisting of a few million parameters to today's ultra-large-scale models with billions or even hundreds of billions of parameters, model complexity and computational requirements have grown exponentially. To accelerate model training, distributed training techniques have been widely used, including parallel strategies in various dimensions, such as model parallelism, data parallelism, pipeline parallelism, and expert parallelism. In distributed training, multiple computing nodes (servers) work together to synchronize model parameters and gradient information through collective communication. Collective communication is a core component of distributed training, and its performance and stability directly impact the efficiency and results of model training.

[0003] In existing distributed training systems, collective communication is usually implemented by specialized communication libraries (such as NCCL, MPI, etc.). These communication libraries provide efficient communication primitives (such as AllReduce, Broadcast, AllGather, etc.) for synchronizing and computing data between multiple nodes. However, various failures may occur during collective communication, such as: Communication timeout: Due to network latency or insufficient bandwidth, some communication tasks may not be completed within the expected time. Communication failure: Due to hardware failure, software error or network problem, the communication task may fail directly. Performance bottleneck: The inefficiency of collective communication may cause a significant decrease in the model training speed. Intermittent failure: Some failures may only occur under specific conditions and are difficult to detect by conventional means.

[0004] Although existing distributed training frameworks (such as PyTorch) provide certain fault detection mechanisms, these mechanisms can usually only detect obvious errors and cannot deeply analyze abnormal situations in collective communications. For example: Lack of fine-grained monitoring: Existing systems can usually only detect the overall status of communication tasks, but cannot obtain detailed counting information for each communication primitive. Limited fault diagnosis capabilities: After detecting a fault, existing systems can often only provide simple error information and cannot accurately analyze the root cause of the fault. Difficult to adapt to complex scenarios: In large-scale distributed training, multiple servers and multiple RANKs are involved. RANK is an index number for processes in model training. In model training, one graphics card usually corresponds to one RANK number. Existing systems find it difficult to effectively coordinate and analyze communication failures across nodes. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the present invention aims to provide a model training fault diagnosis method and system for collective communication.

[0006] The object of the present invention is achieved through the following technical solution: a model training fault diagnosis method for collective communication, comprising the following steps:

[0007] After receiving the model training task, a counting function is embedded in the torch_plugin plug-in to collect information about different primitives called during model training and to count the number of collective communication primitives issued by the upper-layer software for model training.

[0008] Add a completion count for processing collective communication primitive tasks to the lower-level communication library and register a query interface with torch_plugin; the completion count includes the number of collective communication primitives completed and the amount of data sent and received during the actual collective communication process;

[0009] For different processes in the model training process, separate fault diagnosis and analysis threads are started in advance. According to the preset sampling interval, the number of collective communication primitives sent by the upper-layer software, the number of completed collective communication primitives counted in the lower-layer communication library, and the amount of data sent and received during the actual collective communication process are collected through the distributed system. The fault information diagnosis when the model training is abnormal is analyzed.

[0010] Furthermore, the communication library includes NCCL, GLOO, MPI and other back-end versions of the communication library.

[0011] Furthermore, the different processes include: a main process (RANK0), which is used to receive data transmitted by other processes and perform distributed data collection; other processes (RANK1, RANK2, etc.), which are used to collect data and send it to the main process; the data includes the number of times collective communication primitives are issued, the number of times collective communication primitives have been completed, and the amount of data sent and received during the actual collective communication process.

[0012] Furthermore, the fault information diagnosis when the analysis obtains model training abnormality includes:

[0013] If the collective task count completed by some RANKs in a certain collective communication task is less than that of other RANKs, the node information stuck in training can be obtained; if the collective communication operation of a node does not match the number of training iterations, it is determined whether the parameter update of certain layers in the model structure causes additional communication overhead; if the primitive count is normal but the amount of data sent and received is too large, it is determined whether retransmission occurs during data transmission.

[0014] The present invention also provides a collective communication-oriented model training fault diagnosis system, comprising:

[0015] The collective communication primitive task counting module is used to count the number of collective communication primitives issued by the upper-layer software during model training;

[0016] The collective communication layer completes primitive counting module, which is used to count the number of times the lower layer communication library has completed the collective communication primitive and the amount of data sent and received during the actual collective communication process;

[0017] The distributed collection and transmission module is used to collect and transmit the triggering times of collective communication primitives obtained by the collective communication primitive task counting module and the number of completed collective communication primitives obtained by the collective communication layer completion primitive counting module in different processes during the model training process, as well as the amount of data sent and received during the actual collective communication process. On the main RANK, this module implements the distributed collection function. For other RANKs, this module sends the organized data packets to the central node.

[0018] This module uses a socket streaming connection. By using a management network card, it avoids the problem of RDMA (Remote Direct Memory Access) high-speed network card failure during training, which can prevent the central node from collecting data. This ensures the stability of the connection and transmission. A socket connection is an intermediate software abstraction layer between the application layer and the TCP / IP protocol suite. It is a communication mechanism in computer networks that allows different computers to communicate over the network.

[0019] The fault analysis module is used to obtain fault information diagnosis when model training is abnormal from data analysis; the data analysis includes the number of times the upper-layer software sends collective communication primitives, the number of times the lower-layer communication library has completed collective communication primitives, and the amount of data sent and received during the actual collective communication process.

[0020] Furthermore, the collective communication layer completes the primitive counting module and supports multiple communication backends registered to the PyTorch framework in the torch_plugin mode.

[0021] Furthermore, the distributed information collection and sending module collects counting information according to the set sampling interval through a separate fault diagnosis and analysis thread that is started in advance; the technical information includes the number of times the upper-layer software sends down the collective communication primitives, the number of times the lower-layer communication library has completed the collective communication primitives, and the amount of data sent and received during the actual collective communication process.

[0022] Furthermore, the system includes a data retransmission module for periodically checking local temporary files. If the connection status with the central node is abnormal, the data is read from the temporary files and resent. The temporary files are files temporarily stored locally after other RANKs fail to send data to the main RANK central node. The network transmission reliability for distributed scenarios is improved. By means of temporary files, data packets that failed to be sent are cached in the event of network anomalies.

[0023] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the model training fault diagnosis method for collective communication.

[0024] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the model training fault diagnosis method for collective communication is implemented.

[0025] The beneficial effects of the present invention are as follows: the present invention obtains detailed collective communication task information by counting the number of triggering and completion times of collective communication primitives, as well as the actual amount of data sent and received, thereby providing fine-grained monitoring; by analyzing the collective communication task count data, the type and location of communication failures can be quickly identified, thereby achieving rapid fault location. The present invention supports distributed environments and can perform data collection and analysis across multiple servers and multiple ranks. It is suitable for large-scale distributed training scenarios, thereby improving troubleshooting efficiency; by generating detailed fault diagnosis information, it helps developers quickly locate and solve problems, thereby significantly improving the efficiency and reliability of model training tasks.

[0026] Through the above innovations, the present invention can effectively solve the shortcomings of the existing technology in collective communication fault diagnosis in distributed model training, and provide more reliable technical support for deep learning model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A data flow diagram of the collective communication-oriented model training fault diagnosis method and system according to the present invention;

[0029] Figure 2 This is a diagram showing the main modules of different RANKs described in the present invention;

[0030] Figure 3 This is a schematic flow chart of the main RANK according to the present invention;

[0031] Figure 4 This is a flowchart of the fault diagnosis thread of the main RANK according to the present invention;

[0032] Figure 5 is a schematic flow chart of other RANKs according to the present invention;

[0033] Figure 6 This is a flowchart of the fault diagnosis thread of other RANKs described in the present invention. DETAILED DESCRIPTION

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0035] It should be noted that, unless there is any conflict, the features in the following embodiments and implementations may be combined with each other.

[0036] Example 1

[0037] like Figure 1As shown, a model training fault diagnosis method for collective communication provided by the present invention includes: issuing training tasks, starting diagnostic threads (training scripts), calling collective communication primitives in parallel training (training framework), issuing communication task statistics, counting completion counts and data volume sent and received, information feedback and fault cause diagnosis.

[0038] The user sends a model training task to the distributed system. This task includes information such as training data, model architecture, and training parameters. After receiving the training task, the training script starts a diagnostic statistics thread. This thread prepares for distributed data transmission, regularly collects counting information, and performs fault analysis and location.

[0039] In the training framework, collective communication primitives are called to coordinate data exchange between multiple training nodes, ensuring that all nodes synchronize model parameter updates. The torch_plugin plugin inserts code responsible for collecting and analyzing information about communication tasks issued by the training framework.

[0040] The collective communication library is responsible for counting the completion status of communication tasks, including the number of successfully completed tasks and the current status of the amount of data sent and received. This information is crucial for monitoring training progress and identifying potential problems.

[0041] During training, each component sends the collected information back to the training script. This process implicitly includes the distributed node communication process, that is, the aggregation of collected data.

[0042] If a failure or anomaly occurs during training, the training script uses the collected information to diagnose the cause of the failure.

[0043] Example 2

[0044] like Figure 2 As shown in the figure, the module division of different RANKs in the present invention is as follows. Since the main RANK (RANK0) process is responsible for distributed data collection, other RANKs (RANK1, RANK2, etc.) need to send data to the main RANK central node. Therefore, for the two types of RANKs, there are different modules; specifically, they include the following modules:

[0045] Main RANK distributed collection module: completes communication with other RANKs through Socket streaming connection, and receives statistical information packaged from other RANKs.

[0046] Main RANK set counting module: calls the torch_plugin layer's set task distribution statistics function and the completed set communication tasks and data volume status information reported by the set communication library to package and construct the data.

[0047] Main RANK statistical analysis module: aggregates the counting information from other RANKs and the counting of its own RANK, and analyzes the data to obtain the failure factors for model training.

[0048] Other RANKs are sent to the central node module: the communication with the central node RANK0 is completed through the Socket streaming connection, and the packaged data is sent to the central node.

[0049] Other RANK set counting modules: call the torch_plugin layer's set task distribution statistics function and the completed set communication tasks and data volume status information reported by the set communication library to package and construct the data.

[0050] Other RANK data retransmission modules: regularly check the local temporary file, if the connection status with the central node is normal, read the data from the temporary file and resend it.

[0051] Example 3

[0052] like Figure 3 As shown in FIG, the summary process of the main RANK is that, when fault diagnosis is enabled, the fault diagnosis thread is started, statistical information is collected and processed, and finally the diagnosis result is output.

[0053] Start the fault diagnosis thread: The main RANK startup fault diagnosis thread and other RANK startup fault diagnosis threads have different roles. As the server of the Socket, it needs to handle the connection and data transmission and reception with the client. The detailed process is described in Example 4.

[0054] Collect and process statistical information: Aggregate the counting information of its own RANK and the counting information of other RANKs, and determine the fault factors through the counting differences. For example, in a certain collective communication task, the collective task count completed by some RANKs is less than that of other RANKs, and the nodes stuck in training are obtained; if the collective communication operations of some nodes do not match the number of training iterations, determine whether the additional communication overhead is caused by the parameter update of certain layers of the model structure; if the primitive count is normal but the amount of data sent and received is too large, whether retransmission occurs during data transmission, etc.

[0055] Output diagnostic results: Package statistical information and analysis factors into a report and output it to the user.

[0056] Example 4

[0057] like Figure 4 As shown in FIG, the process of the fault diagnosis thread of the main RANK is described in detail, which describes the connection and data receiving and sending process between the main RANK and other RANKs, and undertakes the task counting and data analysis steps.

[0058] First, the master RANK starts a fault diagnosis thread, creates a socket and binds it to the specified IP and port, listens for client connections, waits for the client to establish a connection, and then creates a new thread to handle data transmission and reception with the client. When receiving data from the client, it parses the received packet into a dictionary format, extracting the rank ID, task count, completion count, and sampling event information. Before calling the master RANK statistical analysis module, it calls the master RANK aggregate counting module to obtain local RANK data. Finally, when the client opens a connection, the current client's socket connection is closed.

[0059] Example 5

[0060] like Figure 5 As shown in FIG, it is the summary process of other RANKs. When the fault diagnosis is turned on, the fault diagnosis thread is started and runs in a loop until the end signal is received.

[0061] The detailed process of other RANKs acting as the client of the Socket connection and starting the fault diagnosis thread will be described in the sixth embodiment.

[0062] Example 6

[0063] like Figure 6 As shown in FIG, the process of the fault diagnosis thread of other RANK is described in detail, and the three main tasks of other RANK in the fault diagnosis thread are described, namely, sending data to the central node, calling other RANK set counting modules, and data retransmission.

[0064] After establishing a socket connection with the primary RANK0 central node, the task counting module is called at the sampling interval to obtain the current collective communication task count. A data packet containing the rank ID, task count, completion count, and sampling time is then constructed and sent to the central node. The data transmission status is determined, and if the transmission fails, the data is temporarily stored in a local file. The data retransmission module periodically checks the connection status between the local temporary file and the central node, reads data from the temporary file, and resends it. Upon successful transmission, the previously sent data in the temporary file is deleted.

[0065] An embodiment of the present invention also provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the model training fault diagnosis method for collective communication described in any embodiment.

[0066] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the model training fault diagnosis method for collective communication described in any embodiment.

[0067] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0068] The various embodiments of the present invention are described in a progressive manner. Similar portions between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so its description is relatively simple. For relevant portions, refer to the description of the method embodiment.

[0069] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A model training fault diagnosis method for collective communication, characterized in that: The following steps are involved: After receiving the model training task, a counting function is embedded in the torch_plugin plug-in to collect information about different primitives called during model training and to count the number of collective communication primitives issued by the upper-layer software for model training. Add a completion count for processing collective communication primitive tasks to the lower-level communication library and register a query interface with torch_plugin; the completion count includes the number of collective communication primitives completed and the amount of data sent and received during the actual collective communication process; For different processes in the model training process, separate fault diagnosis and analysis threads are started in advance. According to the preset sampling interval, the number of collective communication primitives sent by the upper-layer software, the number of completed collective communication primitives counted in the lower-layer communication library, and the amount of data sent and received during the actual collective communication process are collected through the distributed system. The fault information diagnosis when the model training is abnormal is analyzed.

2. The model training fault diagnosis method for collective communication according to claim 1, characterized in that: The communication libraries include NCCL, GLOO and MPI.

3. The model training fault diagnosis method for collective communication according to claim 1, characterized in that: The different processes include: a main process, which is used to receive data transmitted by other processes and perform distributed data collection; other processes, which are used to collect data and send it to the main process; the data includes the number of times collective communication primitives are issued, the number of times collective communication primitives have been completed, and the amount of data sent and received during the actual collective communication process.

4. The model training fault diagnosis method for collective communication according to claim 1, characterized in that: The fault information diagnosis when the analysis obtains model training abnormality includes: If the collective task count completed by some RANKs in a certain collective communication task is less than that of other RANKs, the node information stuck in training can be obtained; if the collective communication operation of a node does not match the number of training iterations, it is determined whether the parameter update of certain layers in the model structure causes additional communication overhead; if the primitive count is normal but the amount of data sent and received is too large, it is determined whether retransmission occurs during data transmission.

5. A model training fault diagnosis system for collective communication, characterized in that: include: The collective communication primitive task counting module is used to count the number of collective communication primitives issued by the upper-layer software during model training; The collective communication layer completes primitive counting module, which is used to count the number of times the lower layer communication library has completed the collective communication primitive and the amount of data sent and received during the actual collective communication process; The distributed collection and sending module is used to collect and send the number of collective communication primitives obtained by the collective communication primitive task counting module in different processes and the number of completed collective communication primitives obtained by the collective communication layer completion primitive counting module during the model training process, as well as the amount of data sent and received during the actual collective communication process; The fault analysis module is used to diagnose fault information when model training is abnormal from data analysis.

6. The model training fault diagnosis system for collective communication according to claim 5, characterized in that: The collective communication layer completes the primitive counting module and supports registration with the communication backend of the PyTorch framework in the torch_plugin mode.

7. The model training fault diagnosis system for collective communication according to claim 5, characterized in that: The distributed information collection and transmission module collects counting information according to a set sampling interval through a separate fault diagnosis and analysis thread started in advance.

8. The collective communication-oriented model training fault diagnosis system according to claim 5, characterized in that: The system includes a data retransmission module for periodically checking a local temporary file, and if the connection status with the central node is normal, reading data from the temporary file and resending it.

9. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein, the memory is used to store program data, and the processor is used to execute the program data to implement the model training fault diagnosis method for collective communication as described in any one of claims 1-4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the model training fault diagnosis method for collective communication according to any one of claims 1 to 4 is implemented.

Images