Ensemble communication method and device, chip product, equipment and storage medium

Abstract

The invention discloses a set communication method and device, a chip product, equipment and a storage medium, and relates to the technical field of distributed training. The method is executed by an adaptation layer, and the adaptation layer reconstructs a working communication process group based on a currently surviving training process under the condition that a fault process exists; an actual communication interface corresponding to the working communication process group is called, set communication operation corresponding to set communication is executed, a set communication result is obtained, the set communication result comprises fragment data of all training processes in the working communication process group, and the fragment data are data used when the training processes execute the distributed training tasks; adjusting operation is executed on the set communication result to obtain an adjusted set communication result, and the adjusting operation is used for reducing precision loss generated by missing fragment data of the fault process; and sending the adjusted set communication result to the training process in the working communication process group. According to the method and the device, the fault-tolerant process of the set communication is not perceived by the training process in a data fragmentation scene.

Description

Technical Field

[0001] This application relates to the field of distributed training technology, and in particular to a collection communication method, device, chip product, equipment and storage medium. Background Technology

[0002] Distributed training is a method for training machine learning models that distributes the training process across multiple computing nodes in a distributed system to achieve parallel training. Each computing node can run one or more processes, and each process can manage one or more GPUs (Graphics Processing Units) to execute distributed training tasks. Processes can communicate with each other using collective communications, which is used to synchronize training state data, such as model parameters and gradients, among multiple processes (i.e., computing nodes), ensuring that all processes function as a whole to accelerate the training of the machine learning model.

[0003] In related technologies, to reduce the memory usage of each computing node during distributed training, data sharding is typically used to store the training state data of the machine learning model. This involves evenly dividing the training state data into multiple shards, with each shard stored on a separate computing node. Before officially starting distributed training, the distributed system divides all processes participating in the distributed training task into multiple communication process groups. For each communication process group, the processes within the group first aggregate the shard data from each process in that group through set communication to obtain complete training state data. Then, based on this complete training state data, subsequent forward computation or backward propagation processes are executed to train the machine learning model.

[0004] However, in related technologies, if a process fails in a communication process group, the other processes in that communication process group will be unable to complete the aggregation of the sharded data, resulting in the failure of the distributed training task, which is not conducive to improving the reliability of the distributed system. Summary of the Invention

[0005] This application provides a collection of communication methods, apparatus, chip products, devices, and storage media. The technical solutions provided by this application include the following.

[0006] According to one aspect of the embodiments of this application, a collective communication method is provided, the method being executed by an adaptation layer located between a training process and an actual communication interface, the training process being used to perform distributed training tasks, and the actual communication interface being used to perform collective communication operations; the method includes: For multiple training processes participating in ensemble communication, if it is determined that there is a faulty process among the multiple training processes, a working communication process group is reconstructed based on the currently surviving training processes. The working communication process group is used to replace the historical communication process group used before the fault. The actual communication interface corresponding to the working communication process group is called to execute the set communication operation corresponding to the set communication, and the set communication result is obtained. The set communication result includes the shard data of each training process in the working communication process group. The shard data of the training process is the data used by the training process when executing the distributed training task. An adjustment operation is performed on the aggregated communication results to obtain adjusted aggregated communication results. The adjustment operation is used to reduce the accuracy loss caused by missing fragment data of the faulty process. The adjusted set communication results are sent to the training process in the working communication process group.

[0007] According to one aspect of the embodiments of this application, a collective communication device is provided, the device being located between a training process and an actual communication interface, the training process being used to perform distributed training tasks, and the actual communication interface being used to perform collective communication operations; the device includes: The communication group reconstruction module is used to rebuild a working communication process group based on the currently surviving training processes when it is determined that there is a faulty process among the multiple training processes participating in the collective communication. The working communication process group is used to replace the historical communication process group used before the fault. The communication result acquisition module is used to call the actual communication interface corresponding to the working communication process group, execute the collection communication operation corresponding to the collection communication, and obtain the collection communication result. The collection communication result includes the shard data of each training process in the working communication process group. The shard data of the training process is the data used by the training process when executing the distributed training task. The communication result adjustment module is used to perform an adjustment operation on the aggregated communication results to obtain the adjusted aggregated communication results. The adjustment operation is used to reduce the accuracy loss caused by missing fragment data of the faulty process. The communication result sending module is used to send the adjusted set communication results to the training process in the working communication process group.

[0008] According to one aspect of the embodiments of this application, a chip product is provided, wherein a computer program is stored in the chip product, the computer program being loaded and executed by a processor to implement the above-described collection communication method.

[0009] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described collection communication method.

[0010] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, the computer program being loaded and executed by a processor to implement the above-described collection communication method.

[0011] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the above-described collection communication method.

[0012] The technical solutions provided in the embodiments of this application may include the following beneficial effects.

[0013] By deploying an adaptation layer between the training process and the actual communication interface, the adaptation layer reconstructs the working communication process group and calls the actual communication interface of the working communication process group based on the survival status of the training processes participating in the ensemble communication, thereby aggregating the fragmented data of each training process in the working communication process group. This makes the training process unaware of changes to the working communication process group and the actual communication interface, thus enabling the training process to be unaware of the fault-tolerant process of ensemble communication (i.e., aggregating fragmented data).

[0014] Furthermore, in data sharding scenarios, by reconstructing the working communication process group based on the survival status of the training processes participating in ensemble communication, the aggregation of sharded data can continue. This allows surviving training processes to continue executing distributed training tasks, ensuring that the distributed training tasks are not interrupted. Additionally, by adjusting the ensemble communication results, the accuracy loss caused by sharded data from failed processes can be reduced, thus helping to maintain the execution accuracy of distributed training tasks and improving the reliability and fault tolerance of the distributed training system. Simultaneously, since there is no need to wait for fault recovery before ensemble communication or distributed training task execution, the latency introduced during fault recovery is avoided, thereby helping to maintain the training efficiency of the distributed training system. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of a distributed training system provided in one possible implementation of this application; Figure 2 This is a schematic diagram illustrating data parallelism provided in one possible implementation of this application; Figure 3 This is a schematic diagram of pipeline parallelism provided in one possible implementation of this application; Figure 4 This is a schematic diagram of tensor parallelism provided in one possible implementation of this application; Figure 5 This is a schematic diagram of hybrid parallelism provided in one possible implementation of this application; Figure 6 This is a schematic diagram of the collection communication involved in the data fragmentation technology provided in one possible implementation of this application; Figure 7 This is a schematic diagram of a collection communication interruption provided in one possible implementation of this application; Figure 8 This is a schematic diagram of a collection communication interruption provided in another possible implementation of this application; Figure 9 This is a schematic diagram of the adaptation layer provided in one possible implementation of this application; Figure 10 This is a schematic diagram illustrating uninterrupted collection communication provided in one possible implementation of this application; Figure 11 This is a flowchart of a collection communication method provided in one possible implementation of this application; Figure 12 This is a schematic diagram of a communication process group provided in one possible implementation of this application; Figure 13 This is a schematic diagram illustrating uninterrupted collection communication provided in another possible implementation of this application; Figure 14 This is a schematic diagram of a method for obtaining collection communication results provided in one possible implementation of this application; Figure 15 This is a schematic diagram of a method for obtaining collection communication results provided in another possible implementation of this application; Figure 16This is a schematic diagram of a method for obtaining and distributing collection communication results provided in one possible implementation of this application; Figure 17 This is a schematic diagram of the first mapping data provided in one possible implementation of this application; Figure 18 This is a flowchart of a collection communication method provided in another possible implementation of this application; Figure 19 This is a block diagram of a collection communication device provided in one possible implementation of this application; Figure 20 This is a simplified structural block diagram of a computer device provided in one possible implementation of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0018] Please refer to Figure 1 This illustrates a schematic diagram of a distributed training system provided in one possible implementation of this application.

[0019] The distributed training system 100 can refer to a distributed system used for the distributed training of machine learning models. A machine learning model is an algorithm that automatically learns and generates prediction results based on given input data (also called features). For example, a machine learning model can be a neural network model built upon a neural network. Distributed training is a method for training machine learning models by distributing the training process across multiple computing nodes in a distributed system to achieve parallel training and improve the training efficiency of the machine learning model. A distributed system is a system composed of a group of computing nodes that communicate through a network and coordinate their work to complete a common task. In the embodiments of this application, the task used for the distributed training of the machine learning model can be referred to as a distributed training task.

[0020] In one example, the distributed training system 100 may include multiple computing nodes 101. Each computing node 101 can be used to independently process data to provide computing resources. Multiple computing nodes 101 can cooperate to complete the execution of distributed training tasks, such as multiple computing nodes 101 running in parallel.

[0021] Computing node 101 may be implemented as a computer device, such as a server. Exemplarily, computing node 101 may be implemented as at least one of the following: a server equipped with a Central Processing Unit (CPU), a GPU server equipped with a Graphics Processing Unit (GPU), a server equipped with a General-Purpose Computing on Graphics Processing Units (GPGPU), a server equipped with a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a server equipped with a Tensor Processing Unit (TPU), a Field-Programmable Gate Array (FPGA), a server equipped with Tensor Cores, or a server equipped with a Neural Processing Unit (NPU).

[0022] Optionally, the multiple computing nodes 101 can communicate with each other. For example, a communication link is established between the computing nodes 101, enabling them to communicate with each other. This communication link can be implemented through a network, such as a wired network or a wireless network.

[0023] For example, the network architecture of the distributed training system 100 is a leaf-spine network architecture, which includes leaf switches 102 and spine switches 103. Leaf switches 102 can be implemented as Layer 2 switches for connecting computing nodes 101. Spine switches 103 can be implemented as Layer 3 switches responsible for routing and forwarding, and are connected to leaf switches 102. Some computing nodes 101 can form communication links only through leaf switches 102, while others can form communication links simultaneously through leaf switches 102 and spine switches 103. For example, computing nodes H0 and H1 can form a communication link only through leaf switch 0, while computing nodes H0 and H6 can form a communication link through leaf switch 0, spine switch 103, and leaf switch N. This application embodiment does not limit the value of N, which can be set and adjusted according to actual usage requirements. This application embodiment also does not limit the number of switch layers; for example, the distributed training system 100 may also include switches with three or more layers.

[0024] In one example, the distributed training system 100 also includes a master computing node ( Figure 1(Not shown in the diagram), denoted as the Training Master, is responsible for coordinating and scheduling computing nodes 101 and distributed training tasks, enabling multiple computing nodes 101 to collaboratively execute distributed training tasks. The master computing node centrally manages and allocates tasks, ensuring that all computing nodes 101 work collaboratively to optimize training efficiency. A computing node 101 can be randomly selected from the distributed training system 100 as the master computing node, or a separate master computing node can be set up outside the distributed training system 100; this embodiment does not limit this approach.

[0025] Optionally, the master compute node can also be responsible for the monitoring and control of distributed training tasks, the partitioning of sample data for machine learning models, the status monitoring and control of compute node 101, network configuration management, and fault log analysis. In a feasible example, the master compute node can be implemented as a functional module and deployed on compute node 101.

[0026] In one example, each computing node 101 may be equipped with at least one computing device, which includes at least one of the following: CPU, GPU, GPGPU, DSP, ASIC, TPU, FPGA, Tensor Core, and NPU. For example, if computing node 101 is implemented as a GPU server, multiple GPUs may be configured on computing node 101, and each GPU may be implemented as a GPU card (i.e., a chip), which is the computing device. Each computing node 101 may run a process to utilize at least one computing device on computing node 101 to accelerate computing tasks. In the embodiments of this application, the process used to execute distributed training tasks may be called a training process. The training process is the core execution unit that carries the distributed training task. It is responsible for independently completing a series of key operations on a single computing node 101, from data loading, forward computation, backpropagation to gradient computation, and cooperating with other training processes to jointly complete the distributed training task of the entire machine learning model. Optionally, computing node 101 may also be implemented as the above-mentioned computing device, which is not limited in this embodiment of the application.

[0027] For example, each computing node 101 may run one or more training processes, and each training process may manage one or more computing devices. For instance, taking a computing node 101 as a GPU server, the GPU server may include multiple GPUs (i.e., multiple graphics cards or GPU cards), and each GPU server may run multiple training processes. Each training process may manage at least one GPU to perform distributed training tasks. This embodiment of the application illustrates an example where each training process corresponds to one GPU.

[0028] In one example, distributed training of a machine learning model is an iterative process, with each iteration corresponding to a distributed training task in one step. The distributed training system 100 completes the iterative training of the machine learning model by sequentially executing the distributed training tasks in each step. Distributed training typically requires multiple parallel strategies to improve training efficiency; based on the degree of parallelism of different parallel technologies, the distributed training task can be divided into subtasks with varying degrees of parallelism.

[0029] For example, the parallel strategies used in distributed training may include at least one of the following: (1) Data Parallel (DP) Strategy: Multiple computing nodes 101 are divided into multiple computing node groups (i.e., training process groups), and the sample data used to train the machine learning model is split into multiple data subsets. Different data subsets are assigned to different computing node groups, and each computing node group has the same complete model parameters. Each computing node group includes the same number of computing nodes, and each computing node group includes at least one computing node. For example, refer to... Figure 2 The sample data 201 was divided into two subsets, each processed by a separate set of compute nodes to achieve distributed training of the machine learning model. Compute node set 1 and compute node set 2 executed the distributed training task in parallel.

[0030] In a feasible example, a data parallelism strategy can also be adopted with computing devices as the smallest unit, such as dividing the computing devices into multiple computing device groups (i.e., training process groups), and multiple computing device groups executing distributed training tasks in parallel. In this case, the computing device group is equivalent to a computing node group. This application embodiment uses a computing node group as an example for illustration.

[0031] (2) Pipeline Parallel (PP) Strategy: This involves partitioning the machine learning model across layers, distributing the data (such as model parameters) of one part of the model layer across some computing nodes, and distributing the data of another part of the model layer across other computing nodes. The model layer is the foundational data structure of the machine learning model; different model layers can perform different data processing operations, such as convolution, pooling, normalization, linear transformation, and logistic regression. For example, refer to... Figure 3 The model parameters of a portion of the model layers of the machine learning model are assigned to computing node 302, while the model parameters of the remaining model layers are assigned to computing node 303. Computing nodes 302 and 303 complete the distributed training of the machine learning model based on sample data 301.

[0032] (3) Tensor Parallel (TP) strategy: This involves partitioning the machine learning model within each layer, that is, splitting the data (such as model parameters) of the same model layer onto different computation nodes. For example, refer to... Figure 4 For each model layer of the machine learning model, the model parameters of the model layer are evenly distributed across computing nodes 402 and 403. Computing nodes 402 and 403 are simultaneously trained based on sample data 401. PP and TP represent model parallelism.

[0033] (4) Hybrid Parallel (3D Parallel) Strategy: Simultaneously applying DP, PP, and TP. For example, refer to... Figure 5 Taking DP=8, PP=12, and TP=4 as an example, the sample data 501 is split into 8 data subsets according to the parallelism of DP. Each data subset is used to train a complete machine learning model. That is, the multiple computing nodes 101 in the distributed training system 100 are divided into 8 computing node groups, and each computing node group is used to process a data subset. For any complete machine learning model, it is split according to the parallelism of PP, so that different model layers are assigned to different computing nodes 101. For example, taking computing node group 502 as an example, computing node group 502 includes 12 computing device groups. The computing devices (such as GPUs) in each computing device group can be on the same computing node 101 or on different computing nodes 101. The model parameters of the model layers of the machine learning model can be distributed proportionally to the 12 computing device groups.

[0034] For each model layer, it can be further partitioned according to the parallelism of TP, so that different model parameters are assigned to different computing devices (such as GPUs) within the same computing device group. For example, for any computing device group in 12 computing device groups, the model parameters of the corresponding model layer can be divided into 4 parts, and each part can be assigned to one computing device in that computing device group. The values ​​in the above embodiments are merely exemplary and illustrative. This application does not limit the number of computing node groups, the number of computing nodes in computing node groups, the number of computing device groups, or the number of computing devices in computing device groups.

[0035] In one example, during the distributed training of a machine learning model using a data-parallel or hybrid parallel strategy, after each training cycle, each computing node group needs to exchange training state data (such as model parameters, parameter gradients, activation values, etc.) and update the overall model parameters of the machine learning model to complete the entire distributed training task (implemented by the corresponding training process). For instance, before the next training cycle begins, the training state of each computing node group needs to be synchronized to ensure consistency, thereby achieving distributed training of the machine learning model. For example, regarding model parameters, before the next training cycle begins, the updated overall model parameters of the machine learning model will be synchronized to each computing node group to ensure consistency, thus achieving parallel training of the machine learning model parameters.

[0036] The exchange of training state data can be achieved by the training processes corresponding to different computing node groups. Optionally, collective communications can be used to synchronize training state data between each computing node group. Collective communications is a communication mode in distributed systems that can be used to synchronize training state data between multiple computing nodes (i.e., training processes).

[0037] For example, for data parallelism or hybrid parallelism, the distributed training system 100 can use aggregated communication to facilitate communication between training processes, thereby synchronizing training state data among the various computing nodes 101 in the distributed training system 100. This ensures that all computing nodes 101 (training processes) function as a whole to accelerate the training of machine learning models. In other words, the technical solutions provided in this application are applicable to any distributed training system employing a data parallelism strategy or a hybrid parallelism strategy.

[0038] The training processes running on compute node 101 are logical entities participating in ensemble communication. For example, within a communication domain (i.e., a communication process group), each participating training process has a unique rank number. If eight training processes participate in ensemble communication, each process will be assigned one of eight rank numbers from 0 to 7. If a computing device (such as a GPU) is used for ensemble communication, a training process is typically bound to one computing device. If each compute node 101 has multiple computing devices (e.g., eight computing devices), then during distributed training, each compute node 101 will typically have a corresponding number of training processes (e.g., eight), meaning each compute node 101 can correspond to eight different rank numbers. Training processes participating in communication can communicate across compute node groups. Training processes across compute node groups can communicate via a network (or any supported connection method). Training processes within the same compute node 101 can communicate via any supported connection method, such as PCIe (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard), private connections, etc. This embodiment does not limit the specific methods used.

[0039] In one example, to reduce the memory usage of individual computing nodes during distributed training, data sharding is typically used to store the training state data of the machine learning model. Exemplarily, data sharding may include at least one of the following: (1) ZeRO-1 technology.

[0040] The ZeRO-1 technique only performs sharding operations on the optimizer state to distribute it to different groups of computing nodes (i.e., training process groups). During distributed training, ZeRO-1 can use the communication operator Reduce Scatter to reduce gradient parameters to different training processes (i.e., computing devices or computing nodes), and can use the communication operator Allgather to aggregate the updated model parameters distributed across various computing nodes to all computing nodes. Therefore, ZeRO-1 uses two communication operators: Reduce Scatter and Allgather.

[0041] For example, refer to Figure 6For communication process group 600, if the model parameters are bf16, the backpropagation of rank0 (i.e., training process 0) calculates the bf16 gradient parameters, while the main gradients and main weights are both fp32, and the optimizer only updates the fp32 main weights, then the ensemble communication process can include the following: (1) and (2) the backpropagation of rank0 ends, and there are 16 fp32 gradient elements in the gradient buffer. (3) each training process in communication process group 600 calls Reduce Scatter. (4) each training process (represented by rank) only gets the global reduction result of the 4 elements it is responsible for locally, and the remaining 12 positions are invalid data. (5) each training process performs optimizer update on the 4 fp32 main gradients locally to update the corresponding 4 fp32 main weights. (6) each training process copies and converts these 4 fp32 main weights into bf16, and writes them back to the corresponding bf16 model parameter buffer. (7) and (8) Each training process calls the communication operator Allgather to concatenate the four bf16 parameters of each training process into a complete 16-element bf16 parameter vector and broadcast it to all training processes. (9) The parameter tensors in the model have pointed to this bf16 buffer. Once Allgather is completed, the next round of forward computation can start immediately.

[0042] (2) ZeRO-2 technology.

[0043] The ZeRO-2 technique builds upon the ZeRO-1 technique by adding a sharding operation to the parameter gradients, such as sharding the data into gradient shards and optimizer state shards. The gradient reduction in ZeRO-2 still uses Reduce Scatter, and it also uses Allgather to aggregate the updated model parameters distributed across various computing nodes onto all computing nodes.

[0044] (3) ZeRO-3 technology FSDP (Full Shared Data Parallel) technology.

[0045] ZeRO-3 and FSDP technologies employ a "full-sharding" strategy. Building upon ZeRO-2, they add sharding operations on model parameters, including gradient shards, optimizer state shards, and model parameter shards. When using ZeRO-3 and FSDP, the complete model parameters are collected during both forward and backward propagation, a process implemented using the Allgather communication operator. Gradient reduction follows backpropagation using the Reduce Scatter communication operator. After gradient reduction during training, the system only updates its own maintained model parameter shards and optimizer state shards, without requiring additional communication.

[0046] As can be seen from the above, after enabling data sharding, the Reduce Scatter and Allgather communication operators are mainly used for aggregate communication between communication process groups to exchange information such as optimizer status, parameter gradients, and model parameters. For other unsharded data, Allreduce is still used for communication; specific scenarios are not fully described here. Therefore, after enabling data sharding, the communication operators that need to support aggregate communication fault tolerance are: Reduce, Allreduce, Gather, Allgather, and Reduce Scatter. This application's embodiments provide specific descriptions of Gather, Allgather, and Reduce Scatter.

[0047] For the ZeRO-1 technique, it synchronizes the complete model parameters through Gather or Allgather, and synchronizes the parameter gradients or parameter gradient slices through Reduce Scatter or Allreduce.

[0048] For the ZeRO-2 technique, it synchronizes the complete model parameters through Gather or Allgather, and synchronizes the parameter gradient slices through Reduce Scatter.

[0049] For ZeRO-3 and FSDP technologies, they aggregate fragmented data (such as model parameter fragments) through Gather or Allgather to obtain complete model parameters, and determine and distribute updated fragmented data (such as updated parameter gradient fragments) through Reduce Scatter.

[0050] In related technologies, ensemble communication requires initializing the communication process group at the start of distributed training according to different parallel strategies. The members of the communication process group remain unchanged during training. A communication process group is a set of training processes from different computing node groups. During distributed training, the training processes within a communication process group communicate with each other but not with training processes outside that group. If the parallel strategy and parallelism parameters need to be adjusted, the communication process group needs to be reinitialized for ensemble communication. Furthermore, during ensemble communication, each training process in the communication process group needs to simultaneously call the ensemble communication interface. If any training process fails and cannot participate in communication, the entire ensemble communication is interrupted, leading to the interruption of the entire distributed training task. The performance of ensemble communication directly affects the speed of the distributed training task, determining whether all computing nodes (training processes) in the cluster can work together to accelerate model training. For example, if one training process fails in ensemble communication, the other training processes in the communication process group are in a waiting state, causing ensemble communication to be interrupted, and consequently, the distributed training to be interrupted.

[0051] In data sharding scenarios, training processes within a communication process group need to first aggregate the sharded data from each training process in that group via ensemble communication to obtain complete training state data. Then, based on this complete training state data, subsequent forward computation or backpropagation processes are executed to train the machine learning model. However, if a process fails within a communication process group, since the members of the group remain constant, the training processes in that group will be unable to complete ensemble communication. This prevents the training processes from aggregating the sharded data, leading to the failure of the distributed training task, which is detrimental to improving the reliability of the distributed system.

[0052] In one example, we take the Ring-AllReduce (Ring-AllReduce) aggregate communication method. Ring-AllReduce groups the training processes within the same communication process group into a ring-shaped communication topology. Each training process exchanges data only with its two adjacent training processes. Data from each training process is passed sequentially according to the ring topology, completing AllReduce computations (such as summation, maximum value, average, etc.). In the case of Ring-AllReduce, the communication topology of the communication process group is a ring structure. When the number of training processes in the communication process group is m, the m-th training process communicates with the first training process, and the i-th training process communicates with the (i-1)-th and (i+1)-th training processes, where i is an integer greater than 1 and less than m.

[0053] like Figure 7 As shown, the communication process group 700 includes 4 training processes. The communication topology of the communication process group 700 is a ring structure. The 4 training processes are connected end to end in sequence. The previous training process only transmits data to the next training process. For example, training process 0 only transmits data to training process 1, training process 1 only transmits data to training process 2, training process 2 only transmits data to training process 3, and training process 3 only transmits data to training process 0.

[0054] If training process 3 fails, the communication topology is interrupted by training process 3, causing training processes 0, 1, and 2 to all be in a waiting state. The entire aggregate communication is interrupted, and the data from the four training processes cannot be aggregated. Furthermore, since the next communication target for each training process is fixed when the communication topology is created, training process 2 cannot bypass training process 3 to transfer data if training process 3 fails.

[0055] like Figure 8 As shown, when using FSDP technology, different training processes in communication process group 700 need to perform forward computation or backpropagation after aggregating model parameter shards. In step 2, if training process 3 fails (e.g., the computing device or node where training process 3 resides fails), the aggregate communication corresponding to communication process group 700 is interrupted, resulting in the inability to aggregate model parameter shards and consequently, the interruption of distributed training. After training process 3 recovers from the fault, communication process group 700 can restart distributed training from step 2.

[0056] This application provides a method for addressing the problem of fragmented data aggregation failure due to training process failure causing ensemble communication interruption. The method deploys an adaptation layer between the training process and the actual communication interface. This layer dynamically updates the members of the working communication process group based on the survival status of the training processes participating in ensemble communication, thereby reconstructing the working communication process group. In the event of a failed process, the currently surviving training processes can continue to perform ensemble communication operations, thus avoiding ensemble communication interruption. This ensures that fragmented data aggregation and distributed training tasks are not interrupted. Furthermore, by adjusting the ensemble communication results, the accuracy loss caused by fragmented data from failed processes can be reduced, which helps maintain the execution accuracy of distributed training tasks and improves the reliability and fault tolerance of the distributed training system. Simultaneously, since there is no need to wait for fault recovery before performing ensemble communication to aggregate fragmented data, the latency introduced during fault recovery is avoided, thus helping to maintain the training efficiency of the distributed training system.

[0057] In addition, since the training process only interacts with the adaptation layer, and the adaptation layer reconstructs the working communication process group based on the survival status of the training processes participating in the ensemble communication, and calls the actual communication interface of the working communication process group to obtain the ensemble communication interface, in order to aggregate the fragmented data of each training process in the working communication process group, the training process is unaware of the changes to the working communication process group and the actual communication interface, thereby achieving the training process being unaware of the fault-tolerant process of ensemble communication (i.e., aggregating fragmented data).

[0058] In one example, the aforementioned adaptation layer sits between the training process and the actual communication interface. The training process executes distributed training tasks, while the actual communication interface performs aggregated communication operations. The actual communication interface is the aggregated communication interface (e.g., the communication process group interface) corresponding to the working communication process group. The training process can call the aggregated communication interface corresponding to the adaptation layer to initiate an aggregated communication request. The adaptation layer, in response to the aggregated communication request, can call the actual communication interface corresponding to the working communication process group to perform the actual aggregated communication operation for each training process within the working communication process group. For example, when the training process calls the aggregated communication interface of the adaptation layer, it can send the identifier information of the proxy communication process group and the data to be communicated (e.g., training status data) to the aggregated communication interface (e.g., the communication process group interface). The adaptation layer processes the sharded data to be synchronized by calling the actual communication interface of the working communication process group, thus obtaining the aggregated communication result.

[0059] The Communication Process Group Interface (CPM) is used for collective communication within a group of communication processes. For example, the CPM is an abstraction layer and programming framework for managing communication process groups, encapsulating and scheduling the underlying collective communication interface. It can be used to organize training processes and manage the communication context, defining how a group of training processes can be organized into a cooperating unit.

[0060] The communication process group interface establishes a background communication thread for each communication process group. The training thread (computation thread) is responsible for forward and backward propagation computation, while the communication thread handles collection communication tasks in the background. The two collaborate asynchronously through a queue, thus achieving overlap between computation and communication. The Collective Communication Interface is a standardized programming interface that provides functional primitives for data exchange (i.e., collection communication functions). It defines the semantics of data exchange operations (i.e., communication algorithms), such as Broadcast, Reduce, Allreduce, Gather, Allgather, Reduce Scatter, and other communication operators. The collection communication interface can be used to call collection communication functions, thereby enabling data exchange between multiple training processes.

[0061] In this embodiment, the adaptation layer can be implemented as a general-purpose collective communication interface for the training process, which can be deployed on each computing node 101 (or computing device) in the distributed training system 100 for the training process to call to achieve collective communication. The adaptation layer can also be implemented as a functional layer in the above-mentioned general-purpose collective communication interface for dynamically reconstructing the working communication process group. This embodiment does not limit the adaptation layer.

[0062] For example, the adapter layer is implemented as a standard communication process group interface, and can achieve fault tolerance for aggregate communication by creating working communication process groups (PCPs) internally for actual communication. The working PCPs are process groups used to actually perform aggregate communication operations. The adapter layer has corresponding proxy communication process groups, which are communication process groups (i.e., initial communication process groups) allocated before the start of distributed training. These proxy communication process groups remain unchanged during distributed training and are accessible to all training processes. The communication topology of the communication process groups can be a ring structure, a tree structure, or other topologies to determine the communication methods and paths between training processes. The communication process group interface can be a wrapper around the aggregate communication backend, such as a wrapper around a low-level dedicated communication library. The aggregate communication functions corresponding to the communication process group interface can be used to perform aggregate communication operations on the communication process groups.

[0063] For example, refer to Figure 9 The adapter layer 900 can provide at least one of the following capabilities: (1) Implement a standard, complete communication process group interface, which may include an object construction interface (constructor) and interfaces for communication operators such as Broadcast, Gather, Reduce, Allreduce, Scatter, Allgather, and Reduce Scatter, so as to maintain interface consistency with upper-layer callers (such as training processes).

[0064] From the caller's perspective, adaptation layer 900 is also a communication process group interface. The parameters created in the communication process group interface include: size (i.e., world size), which is the number of members in the proxy communication process group participating in the collective communication; and rank, the unique identifier of the training process in the proxy communication process group, with values ​​ranging from [0, 1, 2, ..., size-1], maintaining numerical continuity. The communication process group interface corresponding to adaptation layer 900 (i.e., the communication process group interface corresponding to the proxy communication process group) remains unchanged.

[0065] (2) After the proxy communication process group is created, its corresponding parameters remain unchanged for the upper layer caller. The training process can always call the communication process group interface of the adaptation layer with the identifier information of the proxy communication process group.

[0066] (3) The adaptation layer 900 dynamically creates and destroys the working communication process group and the actual communication interface 901 of the working communication process group. The actual communication interface 901 is a standard and complete communication process group interface built for the working communication process group.

[0067] (4) The adaptation layer 900 performs a collection communication operation by dynamically calling the actual communication interface 901 of the working communication process group to obtain the collection communication result.

[0068] The working communication process group (such as the actual communication interface 901) may belong to the same operating system process as the proxy communication process group (i.e., the communication process group interface or the adaptation layer 900), or may belong to two independent operating system processes as needed. This application embodiment does not limit this.

[0069] like Figure 10 As shown, when communication process group 700 is a proxy communication process group, in the event of a failure in training process 3, the adaptation layer reconstructs the working communication process group and its actual communication interface based on training processes 0, 1, and 2, while keeping communication process group 700 unchanged. The adaptation layer calls the actual communication interface corresponding to the working communication process group to perform a collective communication operation on each training process within the working communication process group, obtaining the collective communication result. This allows collective communication to still be achieved between training processes 0, 1, and 2, ensuring uninterrupted collective communication and thus uninterrupted distributed training.

[0070] In this embodiment of the application, the distributed training system 100 may further include a central controller ( Figure 1 (Not shown in the diagram). The central controller can be used to detect the liveness status of the training process, and the aforementioned adaptation layer can learn the liveness status of the training process from the central controller. Optionally, a central controller can be deployed for each communication process group, or a central controller can be deployed for all communication process groups. The central controller can be implemented as a functional layer and deployed in the computing node 101 of the distributed training system 100, or in the main computing node, or in a computing node located outside the distributed training system 100. This application embodiment does not limit this.

[0071] In one example, a single distributed training system 100 can be implemented as a training replica. If multiple training replicas are built simultaneously and training state data (such as model parameters, parameter gradients, optimizer states, etc.) are synchronized among the multiple training replicas, multi-replica training can be achieved. The technical solutions provided in the embodiments of this application are applicable to any of the multiple training replicas.

[0072] The following will use method embodiments to describe in detail the technical solutions provided in the embodiments of this application. For content not described in the method embodiments, please refer to the above embodiments, which will not be repeated here.

[0073] Please refer to Figure 11 It illustrates a flowchart of a collection communication method provided in one possible implementation of this application, wherein the execution entity of each step of the method can be... Figure 1 The computing node 101 in the distributed training system 100 shown is an adaptation layer in computing node 101. The method may include at least one of the following steps (1101~1104).

[0074] Step 1101: For multiple training processes participating in the collective communication, if it is determined that there is a faulty process among the multiple training processes, a working communication process group is reconstructed based on the currently surviving training processes. The working communication process group is used to replace the historical communication process group used before the fault.

[0075] In this embodiment, the distributed training system includes multiple computing nodes, and training processes on these nodes execute distributed training tasks. Exemplarily, the distributed training system can employ a data parallel strategy or a hybrid parallel strategy to execute distributed training tasks; this embodiment does not limit this approach. For example, each computing node's training process is used to train a complete machine learning model, and training processes on multiple computing nodes can achieve distributed training of the machine learning model. Optionally, the distributed training system employs data sharding technology to manage the training state data corresponding to the machine learning model, such as the aforementioned ZeRO-1, ZeRO-2, ZeRO-3, and FSDP technologies.

[0076] Before starting the distributed training task, the distributed training system can divide the training processes on multiple computing nodes into at least one communication process group according to a parallel strategy. The training processes in each communication process group can communicate with each other collectively. This at least one communication process group can serve as a proxy communication process group (i.e., the initial communication process group) corresponding to the adaptation layer, and the proxy communication process group remains unchanged during the distributed training process.

[0077] For example, in a data parallelism strategy, multiple computing nodes in a distributed training system are divided into multiple computing node groups. These groups are used to execute distributed training tasks in parallel. The training processes running on the multiple computing nodes are initialized into at least one communication process group (i.e., an initial communication process group and a proxy communication process group). Each training process in a communication process group comes from one of the multiple computing node groups. Each computing node group may include the same number of computing nodes, and each communication process group may include the same number of training processes.

[0078] For example, refer to Figure 12 For the hybrid parallel strategy, the distributed training system divides the computing nodes into n computing node groups. Each computing node group includes K+1 computing nodes 1201. Each computing node 1201 has multiple computing devices 1202 (such as GPU cards). Each computing device 1202 runs a training process, that is, each training process manages one computing device 1202.

[0079] Since there are n computing node groups globally, each communication process group includes n training processes. For any training process in a communication process group, its position within its own computing node group is the same as the positions of other training processes in that communication process group within the computing node groups of other training processes. For example, the first training process on the Kth computer node in each computing node group can form a communication process group. The total number of communication process groups is TP. PP and TP are the number of computing nodes in each computing node group, and PP is the number of computing devices 1202 (i.e., training processes) in each computing node 1201.

[0080] Optionally, the adaptation layer executes step 1101 upon receiving a collective communication request from any of the multiple training processes. This step 1101 is executed before calling the actual communication interface of the historical communication process group (i.e., the working communication process group). The adaptation layer may only begin executing the collective communication operation upon receiving collective communication requests from each training process in the communication process group (e.g., the working communication process group). For example, the adaptation layer may only begin executing the collective communication operation upon receiving collective communication requests from all currently active training processes in the communication process group. Here, the collective communication request is a request used to trigger collective communication. For a training process's collective communication request, it may include at least one of the following: the identification information of the proxy communication process group to which the training process belongs, the identification information of the communication operator, the data to be communicated by the training process, and the identification information of the training process. For a collective communication request sent by the adaptation layer, it includes the identification information of the working communication process group, not the identification information of the proxy communication process group.

[0081] The aforementioned multiple training processes may belong to the same proxy communication process group corresponding to the distributed training system. This application embodiment describes the collective communication of a certain proxy communication process group. The collective communication request sent by the training process includes the identification information of the collective communication operation and the identification information of the proxy communication process group, so that the adaptation layer performs the collective communication operation for the proxy communication process group.

[0082] Optionally, the adaptation layer obtains the survival status of multiple training processes participating in the aggregated communication from the central controller, and determines whether there are any faulty processes among the multiple training processes based on the survival status. The survival status of the training processes is used to indicate whether the training process is inactive, and inactive training processes can be identified as faulty processes. For example, if a training process can run normally, its survival status can be determined as active; if a training process cannot run normally, its survival status can be determined as inactive. For example, in cases where the training process fails, freezes, exits, crashes, or malfunctions, or in cases where the computing device or computing node to which it belongs fails, the survival status of the training process can be determined as inactive. The adaptation layer can obtain the survival status of multiple training processes from the aforementioned central controller.

[0083] The aforementioned set communication operation refers to the set communication operation corresponding to set communication. Set communication operations can be used to aggregate fragmented data. For example, the communication operator corresponding to the set communication operation can be implemented as at least one of the following: Gather, Allgather, or Reduce Scatter. This application does not limit this implementation. Fragmented data belongs to the training state data. For different stages of the distributed training task, fragmented data can correspond to different data in the training state data.

[0084] For example, for ZeRO-1 and ZeRO-2 technologies, the aforementioned ensemble communication operation can correspond to the communication operators Gather and Allgather, used to aggregate model parameters from multiple training processes (i.e., the sharded data is now complete model parameters) to achieve model parameter synchronization across multiple training processes. For ZeRO-3 and FSDP technologies, the aforementioned ensemble communication operation can correspond to the communication operators Gather and Allgather, used to aggregate model parameter shards from multiple training processes (i.e., the sharded data is now complete model parameter shards) to obtain complete model parameters for forward computation or backward propagation. The aforementioned ensemble communication operation can also correspond to the communication operator Reduce Scatter, used to aggregate the local parameter gradient shards of each training process to update the local parameter gradient shards of each training process.

[0085] Collective communication is a synchronization mode in distributed training systems. For example, during distributed training, collective communication can be used to synchronize the parameter gradient shards of various computing nodes (training processes) in the distributed training system, thereby achieving global updates of the machine learning model. In the initial stage of distributed training, collective communication can be used to synchronize the model parameter shards of various computing nodes (training processes) in the distributed training system to achieve the initialization of the distributed training system. Collective communication operation is a process that implements collective communication for training processes within a communication process group.

[0086] In this embodiment of the application, although the adaptation layer declares a proxy communication process group to the outside (such as the training process), it actually uses the working communication process group for collective communication so that the currently surviving training process is unaware of changes in the collective communication.

[0087] The aforementioned faulty processes are training processes that have failed and cannot operate normally. The adaptation layer can identify inactive training processes as faulty processes and currently surviving training processes as healthy processes. Faulty processes cannot continue to participate in aggregate communication, while healthy processes can continue to participate. The working communication process group is the actual communication process group used to execute aggregate communication. That is, the adaptation layer actually performs the aforementioned aggregate communication operation on each training process in the working communication process group to fulfill the aggregate communication request of the training process.

[0088] Distributed training of machine learning models is an iterative process. Before a faulty process is detected, the adaptation layer can directly identify the proxy communication process group as the working communication process group. Upon detecting a faulty process, the adaptation layer reorganizes the currently surviving training processes into the working communication process group. For the current iteration, the working communication process group used in the previous iteration can be referred to as the historical communication process group. For example, for the second iteration, the proxy communication process group (i.e., the initial communication process group) is the historical communication process group.

[0089] In one example, the adaptation layer may update only the communication relationships of the faulty processes within the communication topology of the historical communication process group to obtain the updated communication topology, and then rebuild the working communication process group. For example, the rebuilding process of the working communication process group may include the following:

[0090] S11. Based on the communication topology corresponding to the historical communication process group, determine at least one associated process that has a communication relationship with the faulty process. The communication topology is used to indicate the communication relationship of each process in the historical communication process group in the collective communication.

[0091] Communication relationships can be used to indicate data transfer relationships between training processes. For example, if there is a communication relationship between the first training process and the second training process, the first training process is used to transfer data (such as fragmented data) to the second training process. The communication topology can be a topology formed by the communication relationships between multiple training processes. Optionally, the communication topology of the communication process group can be implemented as a ring structure, tree structure, star structure, mesh structure, and any other topology, which is not limited in this embodiment. Among them, for the tree structure, the training processes are connected hierarchically to form a structure similar to an inverted tree; for the star structure, all training processes are connected to the same training process; for the mesh structure, the training processes are connected by multiple paths to form a structure similar to a fishing net. For ease of explanation and understanding, this embodiment uses a ring structure as an example for illustration.

[0092] For example, the adaptation layer can determine the communication topology of a communication process group (such as a historical communication process group) based on the parallel strategy and aggregate communication method of the distributed training system. For instance, if the parallel strategy of the distributed training system is data parallelism, communication relationships can be established between multiple training processes in the communication process group based on a ring structure, generating the communication topology of the communication process group. The adaptation layer can also obtain the communication topology of the communication process group from the aggregate communication library of the distributed training system; this embodiment does not limit this. The aggregate communication library refers to a library used to implement communication between multiple training processes in distributed training.

[0093] Optionally, in the initial stage of distributed training, the adaptation layer automatically acquires the communication topology of the communication process group. During distributed training, the adaptation layer synchronizes data according to the communication topology of the communication process group, enabling the training processes to jointly complete collective communication. Figure 13As shown, the historical communication process group 1300 includes four training processes: rank0 to rank3. For the forward computation or backpropagation of the distributed training task at each stage, the adaptation layer transmits model parameter slices according to the communication topology of the historical communication process group 1300 until each training process obtains complete model parameters. Each training process then performs forward computation or backpropagation based on the complete model parameters, thereby completing the distributed training of the machine learning model. The communication topology of the historical communication process group 1300 is a ring structure, which can be used to indicate communication relationships between training process 0 (rank0) and training process 1 (rank1), training process 1 (rank1) and training process 2 (rank2), training process 2 (rank2) and training process 3 (rank3), and training process 3 (rank3) and training process 0 (rank0). The adaptation layer sequentially transmits model parameter slices among the four training processes according to the communication topology until each training process obtains complete model parameters.

[0094] Optionally, the adaptation layer can obtain different numbers of associated processes or different associated processes depending on the communication topology of different structures. For example, for a many-to-many communication topology, all training processes except the faulty process can be identified as associated processes; for a many-to-one communication topology, the root process (on which all fragmented data is aggregated) can be identified as the associated process of the faulty process; for a ring-structured communication topology, the previous training process and the next training process of the faulty process can be identified as the associated processes of the faulty process. This application embodiment does not limit this.

[0095] For example, refer to Figure 13 Taking a ring-shaped communication topology as an example, if training process 2 (i.e., rank2) is a faulty process, the adaptation layer can determine training process 1 (i.e., rank1) and training process 3 (i.e., rank3) as associated processes of training process 2 (i.e., rank2).

[0096] S12, for the communication topology, clear the communication relationship between the faulty process and at least one associated process, and adjust the communication relationship between the remaining processes to obtain the updated communication topology.

[0097] The adaptation layer only updates the communication relationship between the faulty process and at least one associated process, and maintains the communication relationship between other training processes (including the remaining processes and associated processes) to reduce the workload of adjusting the communication topology, thereby reducing the workload of rebuilding the working communication process group.

[0098] Optionally, the adaptation layer can directly remove the faulty process from the communication topology, adjust the communication relationship between at least one associated process, and the communication relationship between at least one associated process and the remaining processes, to obtain an updated communication topology. Here, the remaining processes refer to the training processes in the historical communication process group, excluding the faulty process and at least one associated process.

[0099] For example, when the communication topology of the historical communication process group is a ring structure, if the number of processes in the historical communication process group is m, then the m-th training process in the historical communication process group has a communication relationship with the first training process, and the i-th training process in the historical communication process group has a communication relationship with the (i-1)-th training process and the (i+1)-th training process, respectively, where i is an integer greater than 1 and less than m.

[0100] If the i-th training process is a faulty process, for the communication topology, the communication relationships between the i-th training process and the (i-1)-th and (i+1)-th training processes are cleared, and a communication relationship is established between the (i-1)-th and (i+1)-th training processes to obtain the updated communication topology.

[0101] In the case that the first training process is a faulty process, for the communication topology, the adaptation layer clears the communication relationship between the first training process and the second training process and the m-th training process, and establishes a communication relationship between the m-th training process and the second training process, thus obtaining the updated communication topology.

[0102] For example, refer to Figure 13 In the event of a failure in training process 2, the adaptation layer clears the communication relationships between training process 2 and training processes 1 and 3 respectively, and establishes a communication relationship between training process 1 and training process 3, resulting in an updated communication topology. The communication relationship between training process 0 and training process 1 remains unchanged, as does the communication relationship between training process 3 and training process 0.

[0103] In a feasible example, the working communication process group is reconstructed by the central controller in the distributed training system. The method by which the central controller reconstructs the working communication process group is the same as the method by which the adaptation layer reconstructs the working communication process group. After reconstructing the working communication process group, the central controller provides the working communication process group to the adaptation layer.

[0104] This embodiment ensures uninterrupted aggregate communication and avoids introducing delays in fault recovery by removing faulty processes from the communication topology and establishing communication relationships between the remaining processes. This improves the performance of aggregate communication and, consequently, the performance of the distributed training system.

[0105] Furthermore, by making local adjustments to communication relationships based on the communication topology, without making global adjustments, the communication topology can be updated quickly. This helps improve the efficiency of communication topology updates and thus reduces the impact on aggregated communication.

[0106] S13, create a working communication process group based on the updated communication topology.

[0107] The updated communication topology is determined as the communication topology of the working communication process group, and the training processes in the updated communication topology, i.e., all currently alive training processes in the historical communication process group, are combined into the working communication process group.

[0108] Optionally, if it is determined that there are no faulty processes based on the survival status, the historical communication process group is identified as the working communication process group, and the actual communication interface of the historical communication process group is identified as the actual communication interface of the working communication process group.

[0109] S14, Based on the working communication process group, create the actual communication interface for the working communication process group.

[0110] Optionally, the adaptation layer constructs a new set communication interface for the working communication process group using a standard set communication interface construction method, and determines the new set communication interface as the actual communication interface of the working communication process group.

[0111] For example, refer to Figure 9 If the actual communication interface 901 is the actual communication interface of the historical communication process group, the adaptation layer will reconstruct the working communication process group, which includes M training processes, and the actual communication interface 902 of the working communication process group in the event of a failed process in the historical communication process group. The historical communication process group includes N training processes, where M is less than N.

[0112] Optionally, after constructing the actual communication interface 902, the adapter layer can destroy the actual communication interface 901.

[0113] In this embodiment of the application, if the training process fails during a certain iteration, the adaptation layer will no longer perform the set communication operation for the training process. Instead, after the training process is restored or replaced, the set communication operation will be performed for the training process in subsequent stages, thereby achieving uninterrupted set communication and thus uninterrupted distributed training.

[0114] Alternatively, you can first try to recover the faulty process. If the faulty process cannot be recovered, then remove the faulty process directly from the communication topology.

[0115] This embodiment reconstructs the working communication process group based on the communication topology of the historical communication process group, which can ensure that the currently surviving training processes can conduct correct collective communication, thereby improving the performance and stability of the distributed training system.

[0116] Step 1102: Call the actual communication interface corresponding to the working communication process group, execute the set communication operation corresponding to the set communication, and obtain the set communication result. The set communication result includes the sharded data of each training process in the working communication process group. The sharded data of the training process is the data used by the training process when executing the distributed training task.

[0117] Upon receiving a collective communication request from each training process in the working communication process group, the adaptation layer calls the actual communication interface corresponding to the working communication process group. Through the collective communication function (such as a communication operator) corresponding to the actual communication interface, it performs collective communication operations on each training process in the working communication process group, thereby obtaining the fragmented data of each training process in the working communication process group and thus obtaining the collective communication result.

[0118] Optionally, the aforementioned sharded data can be shards of training state data, such as model parameter shards, parameter gradient shards, and optimizer state shards. For example, for ZeRO-3 and FSDP technologies, the sharded data is model parameter shards, and the ensemble communication result can be the model parameters obtained by concatenating the sharded data from each training process, which can be used for forward computation or backpropagation of the machine learning model in the current stage; the sharded data can also be the parameter gradient shards assigned to the training process, used to update the local model parameter shards of the training process. For ZeRO-1 and ZeRO-2 technologies, the sharded data is the complete model parameters, and the ensemble communication result can be the average of the model parameters from each training process, which can be used for the distributed training task in the next stage. This application does not limit this aspect.

[0119] For example, refer to Figure 13 After rebuilding the working communication process group, the adaptation layer calls the actual communication interface of the rebuilt working communication process to perform aggregator communication operations on rank0, rank1, and rank3 according to the communication topology of the working communication process, thereby obtaining the fragmented data of rank0, rank1, and rank3. Training process 2 no longer participates in aggregator communication to avoid interrupting it. Training process 2 also no longer participates in the execution of distributed training tasks. For the communication operators Gather and Allgather, the aggregator communication result is the concatenation of the fragmented data of rank0, rank1, and rank3; for the communication operator Reduce Scatter, the aggregator communication result is the fragmented data redefined based on the local fragmented data of rank0, rank1, and rank3.

[0120] In one example, the process of obtaining the results of a collection of communications may include the following.

[0121] S11. For any training process in the working communication process group, send a set communication request to the actual communication interface. The set communication request includes the identification information of the training process, the identification information of the working communication process group, and the data to be communicated by the training process.

[0122] The set communication request is used to request the set communication function corresponding to the actual communication interface to perform set communication operations. The adaptation layer can obtain the set communication request to be sent to the actual communication interface by adjusting the set communication request of the training process. For example, by adjusting the identifier information of the proxy communication process group in the set communication request of the training process to the identifier information of the working communication process group to which the communication process belongs, the set communication request to be sent to the actual communication interface can be obtained.

[0123] For the communication operators Gather and Allgather, the data to be communicated by the training process is the model parameter slices or model parameters; for the communication operator Reduce Scatter, the data to be communicated by the training process is the parameter gradient slices local to the training process.

[0124] S12, Receive the set communication result returned by the actual communication interface. The set communication result is obtained by the set communication function corresponding to the actual communication interface performing set communication operations.

[0125] The ensemble communication function uses communication operators to calculate the data to be communicated by each training process in the working communication process group, obtains the ensemble communication result, and returns the ensemble communication result to the adaptation layer.

[0126] For example, for the communication operators Gather and Allgather, the aggregate communication result can be obtained by concatenating the fragmented data provided by each training process in the working communication process group. For instance, for the model parameter fragments provided by each training process in the working communication process group, the communication operator corresponding to the actual communication interface is executed to obtain the aggregate communication result. The communication operator corresponding to the actual communication interface can be implemented as at least one of the following: Gather or Allgather.

[0127] For example, refer to Figure 14Historical communication process group 1401 includes 4 training processes: rank0 to rank3. The communication operator corresponding to historical communication process group 1401 is Gather, and rank0 is the root process. In the absence of faulty processes, the communication operator Gather aggregates the model parameter slices of the 4 training processes and concatenates the model parameter slices of the 4 training processes to obtain the set communication result. For example, the communication operator Gather concatenates [1], [2], [3] and [4] into [1, 2, 3, 4]. The final result corresponding to Gather (as shown in the adjusted set communication result below) is only sent to the root process rank0.

[0128] If rank2 fails, the historical communication process group 1401 is updated to the working communication process group 1402. The communication operator Gather aggregates the model parameter slices of rank0, rank1 and rank3, and splices the model parameter slices of the three training processes to obtain the set communication result. For example, the communication operator Gather splices [1], [2] and [4] into [1, 2, 4].

[0129] For example, refer to Figure 15 When the communication operator corresponding to historical communication process group 1401 is Allgather, Allgather aggregates the model parameter slices of the four training processes and concatenates the model parameter slices of the four training processes to obtain the ensemble communication result. Rank 0 to Rank 3 all obtain the ensemble communication result [1, 2, 3, 4]. The final result corresponding to Allgather (as shown in the adjusted ensemble communication result below) is sent to each training process in the working communication process group. If rank 2 does not fail, rank 0, rank 1, rank 2, and rank 3 all obtain the ensemble communication result [1, 2, 3, 4]; if rank 2 fails, rank 0, rank 1, and rank 3 all obtain the ensemble communication result [1, 2, 4].

[0130] For example, for the communication operator Reduce Scatter, the aggregate communication result can be obtained based on the parameter gradient shards provided by each training process in the working communication process group. For instance, for the parameter gradient shards provided by each training process in the working communication process group, the communication operator Reduce Scatter corresponding to the actual communication interface is executed to obtain the aggregate communication result. In the embodiments of this application, for the communication operator Reduce Scatter, the above-mentioned aggregate communication result is the reduction result of the parameter gradient shards provided by each training process, and is not sliced.

[0131] For example, if the operation type of Reduce Scatter is SUM (summation operation), then Reduce Scatter sums the parameter gradient slices sent by each training process in the worker communication process group to obtain the set communication result.

[0132] For example, refer to Figure 16 The historical communication process group 1601 includes four training processes: rank0 to rank3. If the communication operator corresponding to the historical communication process group 1601 is Reduce Scatter, then in the absence of a faulty process, for any of the four training processes, Reduce Scatter can first aggregate the local parameter gradient slices sent by the four training processes, and sum the local parameter gradient slices of the four training processes to obtain the set communication result. For example, Reduce Scatter sums [1], [2], [3] and [4] to obtain the set communication result

[10] . The final result corresponding to Reduce Scatter (as shown in the adjusted set communication result below) is only sent to the root process rank0. The local parameter gradient slice is the parameter gradient slice obtained by the training process in local update.

[0133] If rank2 fails, the historical communication process group 1601 is updated to the working communication process group 1602. Taking rank0 as an example, the Reduce Scatter aggregates the local parameter gradient slices of rank0, rank1 and rank3, and sums the local parameter gradient slices of the three training processes to obtain the set communication result. For example, the Reduce Scatter sums [1], [2] and [4] to obtain the set communication result [7].

[0134] In the event of a faulty process, the embodiments of this application continue to perform the combined communication operation through the actual communication interface of the working communication process, so that the combined communication is uninterrupted and the training process is unaware of the fault tolerance of the combined communication.

[0135] Step 1103: Perform an adjustment operation on the aggregated communication results to obtain the adjusted aggregated communication results. The adjustment operation is used to reduce the accuracy loss caused by missing fragmented data of faulty processes.

[0136] Optionally, the adaptation layer can determine the aggregated communication result returned by the actual communication interface as the adjusted aggregated communication result, or it can calibrate the aggregated communication result returned by the actual communication interface for faulty processes to obtain the adjusted aggregated communication result. This embodiment of the application does not limit this approach. For example, if there is no faulty process in the proxy communication process group, the aggregated communication result returned by the actual communication interface of the proxy communication process group can be determined as the adjusted aggregated communication result; if there is a faulty process in the proxy communication process group, an adjustment operation is performed on the aggregated communication result to obtain the adjusted aggregated communication result.

[0137] Optionally, different classifications of the collective communication operations result in different adjustment operations for the collective communication results. For example, the adaptation layer performs the target adjustment operation corresponding to the collective communication operation on the collective communication result to obtain the adjusted collective communication result; wherein, different classifications of collective communication operations correspond to different target adjustment operations.

[0138] In this embodiment, the classification of collection communication operations can include: data collection operations and data reduction operations. For example, the communication operators Gather and Allgather belong to data collection operations, and the communication operator ReduceScatter belongs to data reduction operations. For data collection operations, the target adjustment operation can be a rearrangement optimization operation, that is, rearranging and optimizing the collected fragmented data; for data reduction operations, the target adjustment operation can be a scaling optimization operation, that is, scaling the reduction result.

[0139] The adjustment process for the set communication results differs for different categories of set communication operations. The following will use Gather, Allgather, and Reduce Scatter as examples to illustrate this.

[0140] In one example, for the communication operators Gather and Allgather, the set communication operation is used to obtain the model parameters of the machine learning model corresponding to the distributed training task. For instance, in the forward computation or backpropagation of the machine learning model corresponding to the distributed training task, the sharded data of the training process is the sharded model parameters stored by the training process for the machine learning model. The set communication operation is used to aggregate the model parameter shards of each training process in the working communication process group to obtain the complete model parameters. Therefore, in the case of reconstructing the working communication process group, the adjustment process of the set communication result may include the following:

[0141] S22, based on the first mapping data, rearrange the fragmented data in the set communication result to obtain the rearranged set communication result. The first mapping data is used to indicate the identification information of each training process in the working communication process group in the initial communication process group and the working communication process group, respectively.

[0142] The aforementioned first mapping data is used to indicate the identification information of each training process in the working communication process group in the proxy communication process group (i.e., the initial communication process group) and the working communication process group, respectively, so as to establish the mapping relationship between the identification information of the training process in the proxy communication process group and the identification information in the working communication process group.

[0143] For example, refer to Figure 17 The proxy communication group process group 1701 (i.e., the initial communication process group) includes n training processes, whose ranks (i.e., identifiers) are [0, 1, 2, ..., n-1], and size = n. Due to the failure of rank1, the working communication process group 1702 is reconstructed. The working communication process group 1702 includes n-1 training processes, whose ranks are [0, 1, 2, ..., n-2], and size = n-1. The first mapping data can then be used to indicate that rank0 in proxy communication group process group 1701 corresponds to rank0 in working communication group 1702, rank2 in proxy communication group process group 1701 corresponds to rank1 in working communication group 1702, rank3 in proxy communication group process group 1701 corresponds to rank3 in working communication group 1702, and so on.

[0144] In one example, if the aggregate communication result is stored in a new output storage unit, execution begins with the step of rearranging the fragmented data in the aggregate communication result according to the first mapping data to obtain the rearranged aggregate communication result; or, if the aggregate communication result is indicated by a view of the original output storage unit, a first optimization operation is performed on the aggregate communication result for the faulty process to obtain the adjusted aggregate communication result.

[0145] For the communication operators Gather and Allgather, each set communication operation has a corresponding output storage unit used to store the set communication results. The original output storage unit is the original output storage unit used to store the set communication results. The new output storage unit is a separate, additionally reconstructed output storage unit from the original one; the new and original output storage units are independent storage spaces and do not share memory. If a new output storage unit is constructed to store the set communication results, the set communication results must be copied to the original output storage unit. During the copying process, the sorting of the fragmented data in the set communication results must be restored to the same sorting as the fragmented data corresponding to the initial communication process group.

[0146] The view of the original output storage unit is not an independent storage space, but rather a different window or interpretation of the original output storage unit. That is, the collective communication result is still stored in the original output storage unit without needing to be copied. If the collective communication result is indicated through the view of the original output storage unit, the adaptation layer directly performs the first optimization operation on the collective communication result for the faulty process to obtain the adjusted collective communication result.

[0147] For example, the communication operators Gather or Allgather typically accept two parameters: input (corresponding to the original input storage unit), which can be a tensor used to provide the input data for the training process, such as containing N elements; and output (corresponding to the original output storage unit), which can be a tensor or an array of multiple tensors used to store the communication results. If k ranks participate in the communication, then the size of this parameter is k. N.

[0148] If a new tensor instance (corresponding to a new output storage unit) is constructed based on the set communication result, the data in the new tensor instance needs to be copied to the original tensor instance (corresponding to the original output storage unit). If only a view is set for the set communication result (i.e., sharing the same tensor), then there is no need to rearrange the fragmented data in the set communication result. The new tensor returned by the View function can be called a view of the original tensor. In this case, the new tensor instance is not an independent copy of the data, but a different window or interpretation of the original tensor instance. The process of copying the data in the new tensor instance to the original tensor instance is the process of rearranging the fragmented data in the set communication result.

[0149] For example, refer to Figure 14In the event of a failure in the rank2 training process, each training process generates a new output, denoted as `new output` (which stores the ensemble communication results). The size of `new output` is three times the size of the input. This `new output` can be a new tensor instance or a view created on top of an existing tensor instance. The method of constructing `new output` determines whether to rearrange the fragmented data in the ensemble communication results.

[0150] Optionally, if `new output` is a new tensor instance, the fragmented data in `new output` needs to be rearranged according to the first mapping data so that each fragment is restored to its original position, and missing fragments are filled in. If `new output` is a view, there is no need to rearrange the fragmented data in the collection communication result.

[0151] For example, if the new output corresponding to the ensemble communication result is a new tensor instance, the model parameter fragments corresponding to rank0 are rearranged to the first position, the model parameter fragments corresponding to rank1 are rearranged to the second position, and the model parameter fragments corresponding to rank3 are rearranged to the fourth position, resulting in the rearranged ensemble communication result. The third position corresponding to the faulty process rank2 is temporarily vacant.

[0152] S23, for the faulty process, perform a first optimization operation on the rearranged set communication result to obtain the set communication result. The first optimization operation includes at least one of the following: filling the historical fragment data of the faulty process into the set communication result, or filling the set communication result with set data for the faulty process.

[0153] The historical fragment data of the faulty process can be the fragment data of the faulty process in the previous stage (such as model parameter fragments). The set data can be set and adjusted according to actual usage requirements, such as 0. The specific implementation of the first optimization operation can also be set and adjusted according to actual usage requirements, and this application embodiment does not limit it in this way.

[0154] For example, refer to Figure 14 For the rearranged set communication result, the third position corresponding to rank2 can be filled with 0 to obtain the adjusted set communication result. The fault-tolerant communication of the communication operators Gather and Allgather will result in some data loss, but since the time proportion of failure scenarios is low, the impact of short-term model parameter not being updated on the overall machine learning model is small. Therefore, this method can be used for set communication fault tolerance.

[0155] This embodiment can reduce the accuracy loss caused by set communication fault tolerance by rearranging and optimizing the set communication results corresponding to the working communication process group, thereby ensuring the performance of the distributed training system.

[0156] Optionally, the role of the ensemble communication results differs depending on the data sharding technique. For example, for ZeRO-1 and ZeRO-2 techniques, the ensemble communication results are used to synchronize model parameters to execute the next stage of distributed training. For ZeRO-3 and FSDP techniques, the ensemble communication results are used to aggregate sharded data for forward computation or backpropagation.

[0157] For example, when using ZeRO-3 or FSDP as the data sharding technology, for the communication operators Gather and Allgather, in the forward computation of the machine learning model corresponding to the distributed training task, the ensemble communication result is the complete model parameters, which are used by the training process to determine the output result of the machine learning model. The output result is the result obtained from the forward computation of the machine learning model, and the output result of the training process can be used to determine the training loss of the machine learning model. After the ensemble communication is completed, each training process only retains its local sharded data (i.e., model parameter shards) and releases non-local sharded data to reclaim GPU memory.

[0158] Optionally, during the backpropagation of the machine learning model corresponding to the distributed training task, the ensemble communication result represents the complete model parameters, which are used by the training process to determine local parameter gradient slices for the machine learning model. Each training process in the working communication process group can compute a local parameter gradient slice. The training process can compute the local parameter gradient slice based on the ensemble communication result (i.e., the complete model parameters) and the training loss of the machine learning model.

[0159] This embodiment supports set communication fault tolerance in forward computation and backward propagation, which helps to improve the applicability of the adaptation layer and thus improves the fault tolerance capability of the distributed training system.

[0160] In one example, for the communication operator Reduce Scatter, the aforementioned aggregate communication operation is used to obtain the parameter gradients of the machine learning model corresponding to the distributed training task. For instance, during the backpropagation process of the machine learning model corresponding to the distributed training task, the sharded data of the training process is the local parameter gradient shard of the training process. The local parameter gradient shard is used to determine the parameter gradients of the machine learning model. The aggregate communication operation is used to aggregate the local parameter gradient shards of each training process in the working communication process group to determine the parameter gradients of the machine learning model. After the parameter gradients are sliced, the adjusted aggregate communication result can be obtained. Therefore, in the case of reconstructing the working communication process group, the adjustment process of the aggregate communication result may include the following:

[0161] S31, according to the operation type corresponding to the set communication operation, perform the second optimization operation on the set communication result to obtain the intermediate set communication result.

[0162] In this embodiment, the set communication result is the parameter gradient of the machine learning model determined by the working communication process group, and the intermediate set communication result is the optimized parameter gradient.

[0163] The operation type is used to indicate the type of operation performed by the communication operator (i.e., the set communication operation), such as summation, arithmetic mean, maximum value, minimum value, bitwise AND, bitwise OR, bitwise XAND, product, etc. This application embodiment does not limit this.

[0164] Optionally, a second optimization operation may be set for each operation type. For example, the optimization process for the set communication result may also include at least one of the following:

[0165] (1) When the operation type is the first type, perform an averaging operation on the set communication result to obtain the basic set communication result; based on the basic set communication result and the number of processes in the initial communication process group corresponding to the working communication process group, obtain the intermediate set communication result. The first type includes: summation operation.

[0166] For example, when the operation type is summation, the intermediate set communication result is obtained by multiplying the basic set communication result by the number of processes in the initial communication process group.

[0167] Optionally, based on the operation type, the adaptation layer can also combine user strategies and data (such as model parameters) to distribute and process the communication results.

[0168] For example, when the operation type is summation, the aggregate communication results can be calibrated according to a user-specified strategy or the following second optimization operation can be referenced: If the data are independent and identically distributed, the product between the aggregate communication results and the scaling factor can be determined as the intermediate aggregate communication results. The scaling factor is the ratio between the number of training processes in the agent communication process group and the number of training processes in the worker communication process group.

[0169] (2) When the operation type is the second type, the set communication result is determined as the intermediate set communication result. The second type includes at least one of the following: calculating the arithmetic mean, calculating the maximum value, calculating the minimum value, bitwise AND, bitwise OR, and bitwise XAND.

[0170] For example, when the operation type is arithmetic mean, since the model parameters are uniformly distributed, the loss of some data will not have a significant impact on the final result. Therefore, the second optimization operation can be omitted from the set communication result, and the set communication result can be directly determined as the intermediate set communication result. The adaptation layer can also calibrate the set communication result according to the user-specified strategy, but this application embodiment does not limit this.

[0171] For example, when the operation type is to find the maximum or minimum value, the adaptation layer may not perform the second optimization operation on the set communication result, or it may calibrate the set communication result according to the user-specified strategy. This application embodiment does not limit this.

[0172] For example, when the operation type is bitwise AND or bitwise XAND, the adapter layer may not perform a second optimization operation on the set communication result, or it may calibrate the set communication result according to a user-specified strategy. This application embodiment does not limit this.

[0173] (3) When the operation type is the third type, the scaling factor is determined according to the number of processes in the working communication process group and the number of processes in the initial communication process group corresponding to the working communication process group; the set communication result is adjusted according to the scaling factor to obtain the intermediate set communication result. The third type includes the product operation.

[0174] The scaling factor is determined by the ratio of the number of training processes in the proxy communication process group to the number of training processes in the worker communication process group. An exponential operation is then performed on the aggregate communication results based on the scaling factor to obtain the intermediate aggregate communication results.

[0175] For example, when the operation type is a product operation, if the model parameters are independent and identically distributed, the adaptation layer can calibrate the set communication results according to the scaling factor, or it can calibrate the set communication results according to the user-specified strategy. This application embodiment does not limit this.

[0176] For example, refer to Figure 16In the case of rank2 failure, due to the lack of rank2 data (e.g., [3]), the adaptation layer cannot obtain accurate data for the failure-free case (i.e.,

[10] ). For the summation operation, the overall mean can be estimated by the mean of the data, that is, first average the set communication results (i.e., [7]), and then scale it according to the original number of training processes (i.e., 4) to obtain the intermediate set communication results: [7] (4 / 3)≈9.33]. Here (4 / 3) is the scaling factor. Although the intermediate set communication result still deviates from the true value

[10] , it is an improvement over the set communication result, which can effectively reduce the accuracy loss that may be caused by the set communication fault tolerance process.

[0177] In distributed training systems, since Reduce Scatter is typically used to calculate the gradient average between node groups (using AVG or SUM operators, and usually averaging again after SUM), the lack of gradient data for calculating node groups usually does not significantly affect the overall gradient result after averaging. Therefore, for the communication operator Reduce Scatter, the set communication result in set communication fault tolerance scenarios has a limited impact on the overall training of the distributed training system.

[0178] Optionally, when reconstructing the worker communication process group, each training process generates a new input, denoted as `new input`, which is three times the size of the output. This `new input` can be a new tensor instance or a view created on an existing tensor. If `new input` is a new tensor instance, the data (local parameter gradient slices) from the original tensor instance needs to be copied to the new tensor instance; if `new input` is only a view (i.e., sharing the same tensor), then copying the data from the original tensor instance is unnecessary.

[0179] For example, refer to Figure 16 In the event of a rank2 training failure, the decision to copy data from the original tensor instance (original input) can be determined based on how the new input is constructed. If the new input is a new tensor instance, the data from the original tensor instance (i.e., local parameter gradient slices) should be copied to the corresponding positions in the new input based on the first mapping data, while ignoring the local parameter gradient slices corresponding to the failed process. If the new input is a view, there is no need to copy data from the original tensor instance.

[0180] For example, when the new input is a new tensor instance, the local parameter gradient slice corresponding to rank0 in the original input is copied to the first position of the new input, the local parameter gradient slice corresponding to rank1 in the original input is copied to the second position of the new input, and the local parameter gradient slice corresponding to rank3 in the original input is copied to the third position of the new input, thus obtaining the new input of the Reduce Scatter.

[0181] Optionally, for the ZeRO-1 technology, the set communication operation (i.e., Reduce Scatter) is used to synchronize parameter gradients or local parameter gradient slices. For the ZeRO-2 technology, the set communication operation (i.e., Reduce Scatter) is also used to synchronize local parameter gradient slices. The set communication process for the ZeRO-1 and ZeRO-2 technologies is the same as described in the above embodiments, and will not be repeated here.

[0182] S32 performs a fragmentation operation on the intermediate set communication results to obtain the adjusted set communication results.

[0183] Optionally, if the number of training processes in the proxy communication process group is m, the intermediate ensemble communication result (i.e., the optimized parameter gradient) can be evenly divided into m data pieces, i.e., m parameter gradient pieces. For example, the above ensemble communication result is the parameter gradient obtained by reducing the local parameter gradient pieces of each training process in the working communication process group, the intermediate ensemble communication result is the optimized parameter gradient, and the adjusted ensemble communication result is the m parameter gradient pieces obtained by evenly dividing the optimized parameter gradient. Each training process is assigned one parameter gradient piece from the m parameter gradient pieces to replace the original local parameter gradient piece.

[0184] This embodiment calibrates the collective communication results corresponding to the working communication process group, thereby reducing the accuracy loss caused by collective communication fault tolerance and ensuring the performance of the distributed training system.

[0185] In addition, this embodiment calibrates the aggregate communication results by using different second optimization operations according to different operation types. This helps to improve the rationality of the calibration of the aggregate communication results, thereby improving the accuracy of the adjusted aggregate communication results.

[0186] Step 1104: Send the adjusted set communication results to the training process in the working communication process group.

[0187] For the communication operators Gather and Allgather, the adaptation layer sends the adjusted aggregate communication results to the target training process in the working communication process group. This means that the individual data fragments in the adjusted aggregate communication results are sent to the target training process as a whole. For example, for Gather, if the target training process is the root process, the adaptation layer can send the adjusted aggregate communication results only to the root process, or it can use the Broadcast operator to synchronize the adjusted aggregate communication results from the root process to other training processes in the working communication process group. For Allgather, the target training process can be any training process in the working communication process group; that is, the adaptation layer distributes the adjusted aggregate communication results to all training processes in the working communication process group. The root process can be any training process in the working communication process group.

[0188] For example, refer to Figure 14 For the communication operator Gather, the adaptation layer only needs to send the adjusted set communication results to the root process rank0. (See reference) Figure 15 For the communication operator Allgather, the adaptation layer needs to distribute the adjusted set communication results to the four training processes in the working communication process group 1401. in0 to in3 are the data slices (such as model parameter slices) corresponding to the four training processes respectively, and Out is the adjusted set communication results.

[0189] Each training process in the working communication process group needs to wait for the ensemble communication to complete to ensure that all training processes have received the necessary data for forward computation, backpropagation, or the next stage of distributed training.

[0190] For the communication operator Reduce Scatter, the adaptation layer distributes the sharded data from the adjusted aggregate communication result to each training process in the working communication process group, such that each training process in the working communication process group receives a shard of data from the adjusted aggregate communication result.

[0191] In one example, after the adaptation layer rebuilds the working communication process group, it will synchronously construct the identification information of the currently surviving training process in the working communication process group based on the working communication process group. The adaptation layer can no longer directly use the identification information of the currently surviving training process in the proxy communication process group to return the adjusted set communication result. To address this issue, step 1104 may also include the following.

[0192] S41, determine the first mapping data based on the working communication process group and the initial communication process group corresponding to the working communication process group. The first mapping data is used to indicate the identification information of each training process in the working communication process group in the historical communication process group and the initial communication process group, respectively.

[0193] The first mapping data is used to indicate the correspondence between the identification information of the currently surviving training process in the working communication process group and its corresponding identification information in the proxy communication process group (i.e., the initial communication process group).

[0194] For any training process in the working communication process group, the identification information corresponding to the training process in the working communication process group can be stored in correspondence with its identification information corresponding to the proxy communication process group to obtain the first mapping data. Alternatively, the mapping data between the proxy communication process group and the historical communication process group can be adjusted according to the working communication process group to obtain the first mapping data, but this embodiment of the application does not limit this.

[0195] For example, refer to Figure 17 A correspondence is established between rank0 in proxy communication group process group 1701 and rank0 in working communication process group 1702; a correspondence is established between rank2 in proxy communication group process group 1701 and rank1 in working communication process group 1702; a correspondence is established between rank3 in proxy communication group process group 1701 and rank3 in working communication process group 1702; and so on, to obtain the first mapping data.

[0196] S42, when the sharded data is model parameter sharded, for the target training process in the working communication process group, the adjusted set communication result is sent to the target training process according to the first mapping data.

[0197] Optionally, the adaptation layer sends the adjusted aggregate communication result to the computing node or computing device where the target training process resides. For example, the adaptation layer determines the target identification information of the target training process in the proxy communication process group based on the first mapping data; and sends the adjusted aggregate communication result to the target training process based on the target identification information. The target identification information indicates the corresponding identification information of the target training process in the proxy communication process group.

[0198] For example, refer to Figure 17 For rank3 in the working communication process group 1702, the identifier information corresponding to rank3 in the proxy communication process group 1701 can be determined as 2 according to the first mapping data. That is, the adaptation layer can send the adjusted collection communication result (i.e. with optimized model parameters) to rank2 in the proxy communication process group 1701.

[0199] S43, when the sharded data is local parameter gradient sharding, for any training process in the working communication process group, according to the first mapping data, the sharded data corresponding to the training process in the adjusted set communication result is sent to the training process.

[0200] The fragmented data in the adjusted set communication result is the parameter gradient fragment (which can be called the updated fragmented data), which is sent to the training process to replace the local parameter gradient fragment of the training process.

[0201] For example, the adaptation layer determines the identification information of the training process in the proxy communication process group based on the first mapping data, and sends the slice data of the training process in the adjusted set communication result to the training process based on the identification information.

[0202] For each training process in the working communication process group, after obtaining the corresponding parameter gradient slices, the training process can update its local model parameter slices based on the updated parameter gradient slices, thus completing the model parameter update of the machine learning model. After each training process in the working communication process group has completed the update of its model parameter slices, the next stage of the distributed training task can begin.

[0203] This embodiment establishes first mapping data, which can meet the backtracking requirements of the identification information of the training process in the working communication process group. This allows the adaptation layer to correctly return the adjusted set communication results to the training process based on the first mapping data, which helps to ensure the accuracy of the distribution of the adjusted set communication results.

[0204] In summary, the technical solution provided in this application deploys an adaptation layer between the training process and the actual communication interface. The adaptation layer reconstructs the working communication process group and calls the actual communication interface of the working communication process group based on the survival status of the training processes participating in the collective communication, thereby aggregating the fragmented data of each training process in the working communication process group. This makes the training process unaware of changes to the working communication process group and the actual communication interface, thus enabling the training process to be unaware of the fault-tolerant process of collective communication (i.e., aggregating fragmented data).

[0205] Furthermore, in data sharding scenarios, by reconstructing the working communication process group based on the survival status of the training processes participating in ensemble communication, the aggregation of sharded data can continue. This allows surviving training processes to continue executing distributed training tasks, ensuring that the distributed training tasks are not interrupted. Additionally, by adjusting the ensemble communication results, the accuracy loss caused by sharded data from failed processes can be reduced, thus helping to maintain the execution accuracy of distributed training tasks and improving the reliability and fault tolerance of the distributed training system. Simultaneously, since there is no need to wait for fault recovery before ensemble communication or distributed training task execution, the latency introduced during fault recovery is avoided, thereby helping to maintain the training efficiency of the distributed training system.

[0206] In this embodiment of the application, the central controller performs real-time detection of the survival status of multiple training processes participating in the collective communication, so that the adaptation layer can promptly identify the faulty process. This embodiment of the application may also include the following.

[0207] S51, under the target condition, sends a status acquisition request to the central controller, which is used to monitor the survival status of the training process.

[0208] A status acquisition request is used to request the liveness status of multiple training processes. For example, a status acquisition request may include the identification information of multiple training processes or the identification information of the agent communication process group to which multiple training processes belong, so that the central controller can accurately provide the liveness status of multiple training processes.

[0209] The target scenario includes at least one of the following: a communication anomaly is detected before the actual communication interface of the historical communication process group is called, a communication anomaly is detected during the collective communication of the historical communication process group, or the collective communication of the historical communication process group fails.

[0210] Communication anomalies can refer to exceptions that cause collective communication to fail, such as the occurrence of a faulty process or network failure. For example, if a faulty process is detected during the collective communication process of a historical communication process group, the adaptation layer will stop the collective communication process of the historical communication process group and reconstruct the working communication process group.

[0211] Collective communication failure can refer to situations such as failing to obtain the collective communication result or obtaining an inaccurate collective communication result. For example, if the adaptation layer determines that collective communication for a historical communication process group has failed, it reconstructs the working communication process group to continue executing the collective communication operation; if the adaptation layer determines that collective communication for a historical communication process group has succeeded, it prepares for the next iteration process without reconstructing the working communication process group.

[0212] For example, the adaptation layer may send a status acquisition request to the central controller before each ensemble communication operation, i.e., before calling the actual communication interface of the historical communication process group. As another example, in the case where distributed training at each stage includes multiple ensemble communications, the adaptation layer may also send a status acquisition request to the central controller only before executing the first ensemble communication operation. The adaptation layer may also send a status acquisition request to the central controller if a communication anomaly is detected during the execution of each ensemble communication operation. Furthermore, the adaptation layer may send a status acquisition request to the central controller if a ensemble communication failure is detected. This embodiment of the application does not limit these possibilities.

[0213] In one example, the central controller monitors the liveness status of training processes in a distributed training system in real time. This process may include the following: the central controller determines the liveness status of multiple training processes in real time, and the liveness status indicates whether a training process is inactive. A faulty process is inactive.

[0214] If the training process is inactive, the adaptation layer can determine that the training process has failed and cannot continue to participate in the distributed training task (including aggregate communication). If the training process is alive, the adaptation layer can determine that the training process has not failed and can continue to participate in the distributed training task. This application does not limit the method for determining the survival status.

[0215] In one example, the central controller can use a heartbeat mechanism to detect the survival status of multiple training processes in real time. For example, the central controller maintains heartbeats with each of the multiple training processes, that is, each training process sends a heartbeat signal to the central controller at a set interval to inform the central controller that it is alive. The set interval can be set and adjusted according to actual usage requirements, and this embodiment does not limit it.

[0216] For example, for the first training process among multiple training processes, the central controller determines that the state of the first training process is inactive if it does not receive a heartbeat signal from the first training process for a set period of time. The first computing node can be any one of the multiple computing nodes. The set period can be set and adjusted according to actual usage requirements, and this embodiment does not limit it.

[0217] For example, refer to Figure 13 The distributed training system also includes a central controller 1301. If the central controller 1301 does not receive a heartbeat signal from the training process 2 for a set period of time, it can determine that the survival status of the training process 2 is inactive.

[0218] In one example, the central controller determines that the first training process is inactive upon receiving an abnormal signal from it. The abnormal signal is used to inform the central controller that a malfunction has occurred.

[0219] For example, refer to Figure 13 If training process 2 encounters a fault, it sends an abnormal signal to the central controller 1301 to inform itself of the fault. Optionally, if there are inactive training processes among multiple training processes, the central controller can determine that there is a faulty process among the multiple training processes and send the member status data to the adaptation layer.

[0220] Optionally, the central controller can maintain a member list to record the training processes participating in the distributed training task. If a faulty process exists among multiple training processes, the central controller removes the faulty process from the member list. If the faulty process recovers, the central controller adds it back to the member list.

[0221] The central controller can generate member status data based on the member list, or it can generate member status data based on the individual survival status of multiple training processes; this embodiment does not limit this. The member status data is used to indicate the survival status of multiple training processes participating in the ensemble communication. Upon receiving a status acquisition request, the central controller sends the member status data to the adaptation layer. The member status data can also be used to indicate the survival status of each training process in the proxy communication process group.

[0222] S52 receives member status data sent by the central controller. The member status data is used to indicate the survival status of multiple training processes participating in the ensemble communication.

[0223] After receiving member status data, the adaptation layer determines the survival status of multiple training processes based on the member status data, and then determines whether there are faulty processes (i.e. inactive training processes) in the historical communication process group, so as to determine whether it is necessary to reconstruct the working communication process group.

[0224] This embodiment uses a central controller to monitor the survival status of the training process in real time, enabling the adaptation layer to achieve fault tolerance for ensemble communication (i.e., reconstruct the working communication process group to ensure uninterrupted ensemble communication), thereby achieving uninterrupted ensemble communication.

[0225] In this embodiment, after the faulty process recovers, the faulty process can be reconstructed as a new process to allow the recovered faulty process to continue participating in the distributed training tasks and aggregate communication in subsequent stages. Alternatively, the faulty process can be directly isolated, or other training processes can be used as new processes to replace the faulty process in the reconstruction of the working communication process group. This is beneficial for quickly recovering the aggregate communication in this stage, and this embodiment does not limit the scope of this application.

[0226] In one example, after a faulty process is isolated, the distributed training system can call a new process to replace the faulty process and continue executing the distributed training task, thereby ensuring uninterrupted communication and parallelism of the distributed training system, and thus maintaining the performance and efficiency of distributed training. Embodiments of this application may also include the following:

[0227] S61, if a new process is detected to replace the faulty process, a recovery worker process group is created based on the new process, and the recovery worker process group includes the new process.

[0228] The recovery working process group serves as the working communication process group for aggregate communication in subsequent iterations. Optionally, the adaptation layer can insert a new process into the working communication process group for a failed process to obtain a recovery working process group. For example, this process may include: upon detecting a new process to replace the failed process, adjusting the communication topology of the working communication process group based on the new process to obtain an adjusted communication topology. The adjusted communication topology indicates the communication relationships of each training process in the recovery working process group in aggregate communication. The recovery working process group is then constructed based on the adjusted communication topology.

[0229] Optionally, if there is a backup training process, the backup training process can be used as a new process to replace the faulty process; if there is not enough backup training processes, one can wait for a backup training process to appear and then use the backup training process as a new process to replace the faulty process. This application does not limit this.

[0230] In this embodiment, if a failed process occurs during a stage when backup training processes are available, the new process corresponding to that failed process will not participate in the aggregated communication of that stage. Instead, it will participate in the distributed training task and aggregated communication in the next stage, thereby ensuring uninterrupted aggregated communication and thus uninterrupted distributed training. If there are insufficient backup training processes, newly added processes can participate in the aggregated communication of the target stage, which is the first stage the new process joins.

[0231] After a new process participates in the distributed training task, it proactively informs the central controller of its identity information and establishes a heartbeat with the central controller. Once the heartbeat is established, the new process sends heartbeat signals to the central controller at set intervals.

[0232] Optionally, the adaptation layer can update the communication topology based solely on the communication relationships related to the faulty process within the communication topology of the working communication process group, according to the new process, to obtain the adjusted communication topology. For example, the process of obtaining the adjusted communication topology may include the following:

[0233] (1) Based on the communication topology, identify at least one associated process that has a communication relationship with the faulty process.

[0234] Optionally, the adaptation layer can directly record at least one associated process of the faulty process. The method for determining the associated process is the same as described in the above embodiments, and will not be repeated here.

[0235] (2) For the communication topology of the working communication process group, according to the communication relationship between the faulty process and at least one associated process, establish the communication relationship between the new process and at least one associated process to obtain the adjusted communication topology.

[0236] Optionally, for the communication topology of the working communication process group, the adaptation layer can directly replace the faulty process in the communication topology with a new process to obtain the adjusted communication topology.

[0237] For example, when the communication topology of the working communication process group is a ring structure, and when the i-th training process is a faulty process, for the communication topology of the working communication process group, the adaptation layer clears the communication relationship between the (i-1)-th training process and the (i+1)-th training process, and establishes communication relationships between the new process and the (i-1)-th and (i+1)-th training processes respectively, to obtain the adjusted communication topology.

[0238] In the case that the first training process is a faulty process, for the communication topology of the working communication process group, the adaptation layer clears the communication relationship between the m-th training process and the second training process, establishes a communication relationship between the m-th training process and the new process, and establishes a communication relationship between the new process and the second training process, thus obtaining the adjusted communication topology.

[0239] For example, refer to Figure 13 Upon detecting a new process 2, the adaptation layer clears the communication relationship between rank1 and rank3, establishes a communication relationship between the new process 2 and rank3, and establishes a communication relationship between rank1 and the new process 2, resulting in an adjusted communication topology. The communication relationship between rank0 and rank1 remains unchanged, as does the communication relationship between rank3 and rank0.

[0240] This embodiment adjusts the communication relationships locally based on the communication topology of the working communication process group, without making global adjustments to the communication relationships. This allows for rapid adjustment of the communication topology, which improves the efficiency of the adjustment and reduces the impact on the aggregated communication.

[0241] The adaptation layer determines the adjusted communication topology as the communication topology for the recovery working process group, and constructs the recovery working process group based on the training process indicated by the adjusted communication topology.

[0242] In a feasible example, the adaptation layer creates a recovery worker process group based on the historical communication process group. This could involve replacing faulty processes in the historical communication process group with new processes to obtain the recovery worker process group. Furthermore, the actual communication interface of the historical communication process group is determined as the actual communication interface of the recovery worker process group; alternatively, the actual communication interface of the recovery worker process group is reconstructed.

[0243] This embodiment supports directly replacing faulty processes in historical communication process groups with new processes to build recovery working process groups, which helps improve the efficiency of building recovery working process groups.

[0244] S62, Create the actual communication interface for the recovery worker process group. The set communication function corresponding to the actual communication interface of the recovery worker process group is used to perform set communication operations for the training processes in the recovery worker process group.

[0245] The adaptation layer creates the actual communication interface for the recovery worker process group based on the communication topology of the recovery worker process group. This actual communication interface is implemented as a complete, standard communication process group interface. Optionally, the actual communication interface of the recovery worker process group can be used to perform the aforementioned aggregate communication operations to support the execution of the distributed training task in the current stage.

[0246] The adaptation layer creates a second mapping data between the recovery worker process group and the proxy communication process group. The second mapping data is used to indicate the identification information of the training process in the recovery worker process group and the proxy communication process group, respectively.

[0247] S63, call the actual communication interface corresponding to the recovery worker process group, perform a set communication operation, and obtain the recovery set communication result.

[0248] Upon receiving a collective communication request from each training process in the recovery working process group, the adaptation layer calls the actual communication interface corresponding to the recovery working process group. Through the collective communication function (such as a communication operator) corresponding to the actual communication interface, it performs collective communication operations on each training process in the recovery working process group, thereby achieving data synchronization between the training processes in the recovery working process group.

[0249] When the set communication operation corresponds to the communication operator Gather or Allgather, the recovered set communication result is the set communication result (i.e., model parameters) determined for the recovery worker process group. When the set communication operation corresponds to the communication operator Reduce Scatter, the recovered set communication result is the set communication result (i.e., parameter gradient slices) determined for the recovery worker process group. The process of obtaining the recovered set communication result is the same as the process of obtaining the set communication result described above, and will not be repeated here.

[0250] S64, perform an adjustment operation on the restored set communication result to obtain the adjusted restored set communication result.

[0251] The adjustment process for restoring the set communication results is the same as the adjustment process for the set communication results described above, and will not be repeated here.

[0252] S65 sends the adjusted recovery set communication results to the training process in the recovery worker process group.

[0253] The adaptation layer sends the adjusted recovery set communication results to the training processes in the recovery working process group based on the second mapping data. The distribution process of the adjusted recovery set communication results is the same as the distribution process of the set communication results in the above embodiment, and will not be described again here.

[0254] In this embodiment, during distributed training, the working communication process group is dynamically updated based on the survival status of each training process to maintain uninterrupted ensemble communication and the training scale of the distributed training system until the distributed training task is completed. This helps avoid training state rollback due to training process failures and the resulting loss of computing power, thereby ensuring the performance of the distributed training system.

[0255] In some embodiments, reference Figure 18 Taking FSDP technology as an example, the technical solution provided in this application embodiment may also include the following contents.

[0256] S71, the process before the start of group communication.

[0257] The distributed training system comprises t training processes, which execute distributed training tasks in parallel according to a data parallelism strategy. These t training processes can form a communication process group, which the adaptation layer can designate as a proxy communication process group and declare within the distributed training system. For ease of explanation and understanding, this embodiment uses an example where the training process group consists of only one training process.

[0258] The master compute node divides the model parameters of the machine learning model into t model parameter shards (as the initial model parameter shards, which are updated as distributed training progresses), and each training process corresponds to one of the t model parameter shards.

[0259] After distributed training begins, t training processes perform normal local training on the machine learning model (forward computation, backward computation, parameter gradient computation, etc.).

[0260] S72, the adaptation layer obtains the survival status of the training process.

[0261] Before performing the first set communication for the proxy communication process group, the adaptation layer queries the central controller for the liveness status of t training processes. The first set communication corresponds to the communication operator Allgather, which is used to split the model parameters of each of the t training processes and re-aggregate them into complete model parameters, so that each of the t training processes temporarily has complete model parameters.

[0262] The adaptation layer determines whether the worker process group needs to be rebuilt based on the survival status of the t training processes. For example, if a faulty process is determined to exist, step S73 is executed; if no faulty process is determined to exist, step S74 is executed.

[0263] S73, Rebuild the working communication process group.

[0264] The adaptation layer reconstructs the working communication process group to replace the proxy communication process group based on the surviving training processes in the t training processes, and creates the actual communication interface of the working communication process group.

[0265] After the working communication process group is rebuilt, step S74 is executed.

[0266] S74, perform the first set of communication.

[0267] If no faulty process is found, the proxy communication process group is designated as the working communication process group. The communication operator Allgather is executed to perform the first set of communication by calling the actual communication interface of the proxy communication process group.

[0268] Optionally, in the event of a faulty process, the communication operator Allgather is executed to perform the first set of communication by invoking the actual communication interface of the working communication process group.

[0269] The adaptation layer determines whether the first set communication is successful. If the first set communication is successful, step S75 is executed; if the first set communication fails, step S72 is executed to re-execute the first set communication process for the current working communication process group.

[0270] S75 performs the second set of communication.

[0271] The second set of communication corresponds to the communication operator Reduce Scatter, which is used to reduce the local parameter gradient slices of each of the t training processes to an updated and complete parameter gradient, and then uniformly divide the updated and complete parameter gradient into t parameter gradient slices and distribute them to the t-1 training processes in the working communication process group, with each training process corresponding to one parameter gradient slice.

[0272] The adaptation layer determines whether the second set communication was successful. If the second set communication fails, step S75 is executed again to re-perform the second set communication. This involves first checking the survival status of each training process in the working communication process group to determine whether to reconstruct the working communication process group. If reconstruction is required, the second set communication is performed based on the reconstructed working communication process group. If reconstruction is not required, the second set communication is re-performed based on the working communication process group. If the second set communication is successful, the process jumps to step S76. In other words, for any set communication, if the set communication fails, it is re-executed.

[0273] Optionally, in the event of a failure of the second set of communication, the adaptation layer may also reconstruct the working communication process group and start execution from the first set of communication, which is not limited in this embodiment of the application.

[0274] Optionally, during backpropagation, the training process may perform a first set communication to obtain the complete model parameters and calculate the local parameter gradient slices. After obtaining the local parameter gradient slices, the training process may then perform a second set communication.

[0275] S76, the process after the collection communication is completed.

[0276] After the second set of communication is completed, for each training process in the working communication process group, the training process updates its local shard data according to the assigned parameter gradient shard, such as updating the local model parameter shard, optimizer state shard, and local parameter gradient shard, in preparation for executing the next round of distributed training tasks. The optimizer is used to train the machine learning model, and the optimizer state shard is a shard of the optimizer state.

[0277] After t-1 training processes in the worker communication process group complete their local shard data updates, the master compute node checks whether the current phase of distributed training is complete. If the current phase of distributed training is complete, the master compute node controls all surviving training processes to prepare for the next phase of distributed training; if the current phase of distributed training is not complete, it waits for all surviving training processes to complete their local shard data updates.

[0278] If a faulty process exists, and it recovers during subsequent distributed training, the working communication process group can be reconstructed for that faulty process to execute the distributed training task.

[0279] In summary, by adopting the technical solution provided in this application, an adaptation layer is deployed between the training process and the actual communication interface. The adaptation layer reconstructs the working communication process group and calls the actual communication interface of the working communication process group based on the survival status of the training processes participating in the collective communication, thereby aggregating the fragmented data of each training process in the working communication process group. This makes the training process unaware of changes to the working communication process group and the actual communication interface, thus enabling the training process to be unaware of the fault-tolerant process of collective communication (i.e., aggregating fragmented data).

[0280] Furthermore, in data sharding scenarios, by reconstructing the working communication process group based on the survival status of the training processes participating in ensemble communication, the aggregation of sharded data can continue. This allows surviving training processes to continue executing distributed training tasks, ensuring that the distributed training tasks are not interrupted. Additionally, by adjusting the ensemble communication results, the accuracy loss caused by sharded data from failed processes can be reduced, thus helping to maintain the execution accuracy of distributed training tasks and improving the reliability and fault tolerance of the distributed training system. Simultaneously, since there is no need to wait for fault recovery before ensemble communication or distributed training task execution, the latency introduced during fault recovery is avoided, thereby helping to maintain the training efficiency of the distributed training system.

[0281] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0282] refer to Figure 19 This diagram illustrates a block diagram of a collective communication device provided in one possible implementation of this application. The device has the functionality to implement the method examples described above; this functionality can be implemented in hardware or by hardware executing corresponding software. The device can be the computer device described above, or it can be located within a computer device. The device is situated between the training process and the actual communication interface; the training process is used to execute distributed training tasks, and the actual communication interface is used to execute collective communication operations. Figure 19 As shown, the device 1900 includes: a communication group reconstruction module 1901, a communication result acquisition module 1902, a communication result adjustment module 1903, and a communication result sending module 1904.

[0283] The communication group reconstruction module 1901 is used to reconstruct a working communication process group based on the currently surviving training processes when it is determined that there is a faulty process among the multiple training processes participating in the collective communication. The working communication process group is used to replace the historical communication process group used before the fault.

[0284] The communication result acquisition module 1902 is used to call the actual communication interface corresponding to the working communication process group, execute the collection communication operation corresponding to the collection communication, and obtain the collection communication result. The collection communication result includes the shard data of each training process in the working communication process group. The shard data of the training process is the data used by the training process when executing the distributed training task.

[0285] The communication result adjustment module 1903 is used to perform an adjustment operation on the collective communication result to obtain an adjusted collective communication result. The adjustment operation is used to reduce the accuracy loss caused by missing fragment data of the faulty process.

[0286] The communication result sending module 1904 is used to send the adjusted set communication result to the training process in the working communication process group.

[0287] In some embodiments, the communication result adjustment module 1903 is used to perform a target adjustment operation corresponding to the collective communication operation on the collective communication result to obtain the adjusted collective communication result; wherein, different categories of collective communication operations correspond to different target adjustment operations.

[0288] In some embodiments, the ensemble communication operation is used to obtain the model parameters of the machine learning model corresponding to the distributed training task; the communication result adjustment module 1903 is used to: Based on the first mapping data, the fragmented data in the set communication result is rearranged to obtain the rearranged set communication result. The first mapping data is used to indicate the identification information of each training process in the working communication process group, respectively in the initial communication process group corresponding to the working communication process group and in the working communication process group. For the faulty process, a first optimization operation is performed on the rearranged aggregate communication result to obtain the aggregate communication result. The first optimization operation includes at least one of the following: filling the aggregate communication result with the historical fragment data of the faulty process, or filling the aggregate communication result with set data for the faulty process.

[0289] In some embodiments, the communication result adjustment module 1903 is further configured to: When the set communication result is stored in a new output storage unit, the step of rearranging the fragmented data in the set communication result according to the first mapping data to obtain the rearranged set communication result is executed from the step of rearranging the fragmented data in the set communication result according to the first mapping data. Alternatively, if the aggregate communication result is indicated by the view of the original output storage unit, the first optimization operation is performed on the aggregate communication result for the faulty process to obtain the adjusted aggregate communication result.

[0290] In some embodiments, the ensemble communication operation is used to obtain the parameter gradients of the machine learning model corresponding to the distributed training task; the communication result adjustment module 1903 is used to: Based on the operation type corresponding to the set communication operation, a second optimization operation is performed on the set communication result to obtain an intermediate set communication result; The intermediate set communication results are fragmented to obtain the adjusted set communication results.

[0291] In some embodiments, the communication result adjustment module 1903 is further configured to: When the operation type is the first type, an averaging operation is performed on the set communication results to obtain the basic set communication results; based on the basic set communication results and the number of processes in the initial communication process group corresponding to the working communication process group, the intermediate set communication results are obtained, wherein the first type includes: summation operation; Alternatively, if the operation type is the second type, the set communication result is determined as the intermediate set communication result, where the second type includes at least one of the following: calculating the arithmetic mean, calculating the maximum value, calculating the minimum value, bitwise AND, bitwise OR, and bitwise XAND; Alternatively, if the operation type is the third type, a scaling factor is determined based on the number of processes in the working communication process group and the number of processes in the initial communication process group corresponding to the working communication process group; the set communication result is adjusted according to the scaling factor to obtain the intermediate set communication result, wherein the third type includes a product operation.

[0292] In some embodiments, during the forward computation or backpropagation of the machine learning model corresponding to the distributed training task, the sharded data of the training process is the sharded model parameters stored by the training process for the machine learning model. Alternatively, during the backpropagation process of the machine learning model corresponding to the distributed training task, the shard data of the training process is the local parameter gradient shard of the training process, and the local parameter gradient shard is used to determine the parameter gradient of the machine learning model.

[0293] In some embodiments, the communication result sending module 1904 is configured to: Based on the working communication process group and the initial communication process group corresponding to the working communication process group, first mapping data is determined. The first mapping data is used to indicate the identification information of each training process in the working communication process group in the historical communication process group and the initial communication process group, respectively. In the case where the sharded data is the model parameter shard, for the target training process in the working communication process group, the adjusted set communication result is sent to the target training process according to the first mapping data; When the sharded data is the local parameter gradient shard, for any training process in the working communication process group, according to the first mapping data, the sharded data corresponding to the training process in the adjusted set communication result is sent to the training process.

[0294] In some embodiments, the communication result acquisition module 1902 is configured to: For any training process in the working communication process group, a set communication request is sent to the actual communication interface. The set communication request includes the identification information of the training process, the identification information of the working communication process group, and the data to be communicated by the training process. The system receives the collective communication result returned by the actual communication interface, which is obtained by the collective communication function corresponding to the actual communication interface performing the collective communication operation.

[0295] In some embodiments, the communication group reconstruction module 1901 is further configured to: Under the target condition, a status acquisition request is sent to the central controller, which is used to monitor the survival status of the training process; Receive member status data sent by the central controller, the member status data being used to indicate the survival status of multiple training processes participating in the collective communication; The target scenario includes at least one of the following: a communication anomaly is detected before the actual communication interface of the historical communication process group is invoked, a communication anomaly is detected during the collective communication of the historical communication process group, or the collective communication of the historical communication process group fails.

[0296] In some embodiments, the device 1900 further includes a communication interface creation module (not shown in the figure).

[0297] The communication group reconstruction module 1901 is further configured to, upon detecting a new process for replacing the faulty process, create a recovery working process group based on the new process, the recovery working process group including the new process.

[0298] The communication interface creation module is used to create the actual communication interface of the recovery working process group. The set communication function corresponding to the actual communication interface of the recovery working process group is used to execute the set communication operation for the training process in the recovery working process group.

[0299] The communication result acquisition module 1902 is also used to call the actual communication interface corresponding to the recovery working process group, execute the collection communication operation, and obtain the recovery collection communication result.

[0300] The communication result adjustment module 1903 is further configured to perform an adjustment operation on the recovery set communication result to obtain the adjusted recovery set communication result.

[0301] The communication result sending module 1904 is also used to send the adjusted recovery set communication result to the training process in the recovery working process group.

[0302] In some embodiments, the faulty process is in an inactive state; wherein, for the first training process among the plurality of training processes, the central controller is configured to determine that the state of the first training process is the inactive state if it does not receive a heartbeat signal from the first training process for a set period of time; or, the central controller is configured to determine that the state of the first training process is the inactive state if it receives an abnormal signal from the first training process.

[0303] In some embodiments, the communication group reconstruction module 1901 is configured to: Based on the communication topology corresponding to the historical communication process group, at least one associated process that has a communication relationship with the faulty process is determined, wherein the communication topology is used to indicate the communication relationship of each process in the historical communication process group in the collective communication. For the communication topology, the communication relationship between the faulty process and the at least one associated process is cleared, and the communication relationship between the remaining processes is adjusted to obtain the updated communication topology; Based on the updated communication topology, create the working communication process group; Based on the work communication process group, create the actual communication interface for the work communication process group.

[0304] In summary, the technical solution provided in this application deploys an adaptation layer between the training process and the actual communication interface. The adaptation layer reconstructs the working communication process group and calls the actual communication interface of the working communication process group based on the survival status of the training processes participating in the collective communication, thereby aggregating the fragmented data of each training process in the working communication process group. This makes the training process unaware of changes to the working communication process group and the actual communication interface, thus enabling the training process to be unaware of the fault-tolerant process of collective communication (i.e., aggregating fragmented data).

[0305] Furthermore, in data sharding scenarios, by reconstructing the working communication process group based on the survival status of the training processes participating in ensemble communication, the aggregation of sharded data can continue. This allows surviving training processes to continue executing distributed training tasks, ensuring that the distributed training tasks are not interrupted. Additionally, by adjusting the ensemble communication results, the accuracy loss caused by sharded data from failed processes can be reduced, thus helping to maintain the execution accuracy of distributed training tasks and improving the reliability and fault tolerance of the distributed training system. Simultaneously, since there is no need to wait for fault recovery before ensemble communication or distributed training task execution, the latency introduced during fault recovery is avoided, thereby helping to maintain the training efficiency of the distributed training system.

[0306] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0307] Please refer to Figure 20 This is a simplified structural block diagram of a computer device provided in one possible implementation of this application. The computer device 2000 can be implemented as the aforementioned computing node or master computing node. The computer device 2000 can be used to implement the aggregated communication method provided in the above embodiments.

[0308] Typically, a computer device 2000 includes a processor 2001 and a memory 2002.

[0309] Processor 2001 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 2001 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 2001 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 2001 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 2001 may also include an AI processor, which is used to handle computational operations related to machine learning.

[0310] The memory 2002 may include one or more computer-readable storage media, which may be non-transitory. The memory 2002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 2002 are used to store a computer program configured to be executed by one or more processors to implement the aforementioned collective communication method.

[0311] Those skilled in the art will understand that Figure 20 The structure shown does not constitute a limitation on the computer device 2000, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0312] Optionally, the computer device 2000 may be a server, a server cluster, an artificial intelligence computing cluster, a cloud computing cluster, etc., wherein the artificial intelligence computing cluster may also be referred to as an intelligent computing cluster or a smart computing cluster, and this application does not limit it in this regard.

[0313] In some embodiments, a chip product is also provided, wherein a computer program is stored therein, the computer program being loaded and executed by a processor to implement the above-described collection communication method.

[0314] In some embodiments, a computer-readable storage medium is also provided, wherein a computer program is stored therein, which implements the above-described collection communication method when executed by a processor of a computer device.

[0315] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0316] In some embodiments, a computer program product is also provided, the computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the above-described collection communication method.

[0317] It should be understood that "multiple" as used herein refers to two or more. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order between steps. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this approach.

[0318] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A collective communication method, characterized in that, The method is executed by an adaptation layer, which is located between the training process and the actual communication interface. The training process is used to execute distributed training tasks, and the actual communication interface is used to execute ensemble communication operations. The method includes: For multiple training processes participating in ensemble communication, if it is determined that there is a faulty process among the multiple training processes, a working communication process group is reconstructed based on the currently surviving training processes. The working communication process group is used to replace the historical communication process group used before the fault. The actual communication interface corresponding to the working communication process group is called to execute the set communication operation corresponding to the set communication, and the set communication result is obtained. The set communication result includes the shard data of each training process in the working communication process group. The shard data of the training process is the data used by the training process when executing the distributed training task. An adjustment operation is performed on the aggregated communication results to obtain adjusted aggregated communication results. The adjustment operation is used to reduce the accuracy loss caused by missing fragment data of the faulty process. The adjusted set communication results are sent to the training process in the working communication process group.

2. The method according to claim 1, characterized in that, The step of performing an adjustment operation on the aggregated communication result to obtain the adjusted aggregated communication result includes: Perform the target adjustment operation corresponding to the set communication operation on the set communication result to obtain the adjusted set communication result; Different categories of set communication operations correspond to different target adjustment operations.

3. The method according to claim 2, characterized in that, The set communication operation is used to obtain the model parameters of the machine learning model corresponding to the distributed training task; The step of performing the target adjustment operation corresponding to the collective communication operation on the collective communication result to obtain the adjusted collective communication result includes: Based on the first mapping data, the fragmented data in the set communication result is rearranged to obtain the rearranged set communication result. The first mapping data is used to indicate the identification information of each training process in the working communication process group, respectively in the initial communication process group corresponding to the working communication process group and in the working communication process group. For the faulty process, a first optimization operation is performed on the rearranged aggregate communication result to obtain the adjusted aggregate communication result. The first optimization operation includes at least one of the following: filling the aggregate communication result with the historical fragment data of the faulty process, or filling the aggregate communication result with set data for the faulty process.

4. The method according to claim 3, characterized in that, The method further includes: When the set communication result is stored in a new output storage unit, the step of rearranging the fragmented data in the set communication result according to the first mapping data to obtain the rearranged set communication result is executed from the step of rearranging the fragmented data in the set communication result according to the first mapping data. or, When the aggregate communication result is indicated by the view of the original output storage unit, the first optimization operation is performed on the aggregate communication result for the faulty process to obtain the adjusted aggregate communication result.

5. The method according to claim 2, characterized in that, The set communication operation is used to obtain the parameter gradients of the machine learning model corresponding to the distributed training task; The step of performing the target adjustment operation corresponding to the collective communication operation on the collective communication result to obtain the adjusted collective communication result includes: Based on the operation type corresponding to the set communication operation, a second optimization operation is performed on the set communication result to obtain an intermediate set communication result; The intermediate set communication results are fragmented to obtain the adjusted set communication results.

6. The method according to claim 5, characterized in that, The step of performing a second optimization operation on the set communication result according to the operation type corresponding to the set communication operation to obtain an intermediate set communication result includes: When the operation type is the first type, an averaging operation is performed on the set communication results to obtain the basic set communication results; based on the basic set communication results and the number of processes in the initial communication process group corresponding to the working communication process group, the intermediate set communication results are obtained, wherein the first type includes: summation operation; or, When the operation type is the second type, the set communication result is determined as the intermediate set communication result. The second type includes at least one of the following: calculating the arithmetic mean, calculating the maximum value, calculating the minimum value, bitwise AND, bitwise OR, and bitwise XAND. or, When the operation type is the third type, a scaling factor is determined based on the number of processes in the working communication process group and the number of processes in the initial communication process group corresponding to the working communication process group; the set communication result is adjusted according to the scaling factor to obtain the intermediate set communication result, wherein the third type includes a product operation.

7. The method according to any one of claims 1 to 6, characterized in that, In the forward computation or back propagation of the machine learning model corresponding to the distributed training task, the sharded data of the training process is the sharded model parameters stored by the training process for the machine learning model. or, During the backpropagation process of the machine learning model corresponding to the distributed training task, the shard data of the training process is the local parameter gradient shard of the training process, and the local parameter gradient shard is used to determine the parameter gradient of the machine learning model.

8. The method according to claim 7, characterized in that, Sending the adjusted aggregate communication results to the training process in the working communication process group includes: Based on the working communication process group and the initial communication process group corresponding to the working communication process group, first mapping data is determined. The first mapping data is used to indicate the identification information of each training process in the working communication process group in the historical communication process group and the initial communication process group, respectively. In the case where the sharded data is the model parameter shard, for the target training process in the working communication process group, the adjusted set communication result is sent to the target training process according to the first mapping data; When the sharded data is the local parameter gradient shard, for any training process in the working communication process group, according to the first mapping data, the sharded data corresponding to the training process in the adjusted set communication result is sent to the training process.

9. The method according to any one of claims 1 to 6, characterized in that, The step of calling the actual communication interface corresponding to the working communication process group, executing the set communication operation corresponding to the set communication, and obtaining the set communication result includes: For any training process in the working communication process group, a set communication request is sent to the actual communication interface. The set communication request includes the identification information of the training process, the identification information of the working communication process group, and the data to be communicated by the training process. The system receives the collective communication result returned by the actual communication interface, which is obtained by the collective communication function corresponding to the actual communication interface performing the collective communication operation.

10. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Under the target condition, a status acquisition request is sent to the central controller, which is used to monitor the survival status of the training process; Receive member status data sent by the central controller, the member status data being used to indicate the survival status of multiple training processes participating in the collective communication; The target scenario includes at least one of the following: a communication anomaly is detected before the actual communication interface of the historical communication process group is invoked, a communication anomaly is detected during the collective communication of the historical communication process group, or the collective communication of the historical communication process group fails.

11. The method according to claim 10, characterized in that, The method further includes: If a new process is detected to replace the faulty process, a recovery process group is created based on the new process, the recovery process group including the new process; Create the actual communication interface of the recovery worker process group, and the set communication function corresponding to the actual communication interface of the recovery worker process group is used to execute the set communication operation for the training process in the recovery worker process group; The actual communication interface corresponding to the recovery worker process group is invoked to perform the collection communication operation and obtain the recovery collection communication result; An adjustment operation is performed on the recovered set communication result to obtain the adjusted recovered set communication result; The adjusted recovery set communication results are sent to the training process in the recovery worker process group.

12. The method according to claim 11, characterized in that, The faulty process is inactive; Specifically, for the first training process among the plurality of training processes, the central controller is configured to determine the state of the first training process as the inactive state if it does not receive a heartbeat signal from the first training process for a set period of time; or, the central controller is configured to determine the state of the first training process as the inactive state if it receives an abnormal signal from the first training process.

13. The method according to any one of claims 1 to 6, characterized in that, The reconstruction of the working communication process group based on the currently surviving training processes includes: Based on the communication topology corresponding to the historical communication process group, at least one associated process that has a communication relationship with the faulty process is determined, wherein the communication topology is used to indicate the communication relationship of each process in the historical communication process group in the collective communication. For the communication topology, the communication relationship between the faulty process and the at least one associated process is cleared, and the communication relationship between the remaining processes is adjusted to obtain the updated communication topology; Based on the updated communication topology, create the working communication process group; Based on the work communication process group, create the actual communication interface for the work communication process group.

14. A collective communication device, characterized in that, The device is located between the training process and the actual communication interface, wherein the training process is used to execute distributed training tasks, and the actual communication interface is used to execute aggregate communication operations; the device includes: The communication group reconstruction module is used to rebuild a working communication process group based on the currently surviving training processes when it is determined that there is a faulty process among the multiple training processes participating in the collective communication. The working communication process group is used to replace the historical communication process group used before the fault. The communication result acquisition module is used to call the actual communication interface corresponding to the working communication process group, execute the collection communication operation corresponding to the collection communication, and obtain the collection communication result. The collection communication result includes the shard data of each training process in the working communication process group. The shard data of the training process is the data used by the training process when executing the distributed training task. The communication result adjustment module is used to perform an adjustment operation on the aggregated communication results to obtain the adjusted aggregated communication results. The adjustment operation is used to reduce the accuracy loss caused by missing fragment data of the faulty process. The communication result sending module is used to send the adjusted set communication results to the training process in the working communication process group.

15. A chip product, characterized in that, The chip product stores a computer program, which is loaded and executed by a processor to implement the method as described in any one of claims 1 to 13.

16. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the method as claimed in any one of claims 1 to 13.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the method as described in any one of claims 1 to 13.

18. A computer program product, characterized in that, The computer program product includes a computer program that is executed by a processor to implement the method as claimed in any one of claims 1 to 13.

Images