Ensemble communication method and device, chip product, equipment and storage medium

Abstract

The invention discloses a set communication method and device, a chip product, equipment and a storage medium, and relates to the technical field of distributed training. The method is executed by an adaptation layer, the adaptation layer is located between a training process and an actual communication interface, the training process is used for executing a distributed training task, and the actual communication interface is used for executing a set communication operation; the adaptation layer obtains survival states of a plurality of training processes participating in set communication; under the condition that the existence of the fault process is determined according to the survival state, a working communication process group is reconstructed based on the current survival training process, and the working communication process group is used for replacing a historical communication process group used before the fault; calling an actual communication interface corresponding to the working communication process group, and executing a set communication operation to obtain a set communication result; and sending the set communication result to the training process in the working communication process group. According to the invention, the continuous set communication can be realized, and the training process has no perception on the fault-tolerant process of the set communication.

Description

Technical Field

[0001] This application relates to the field of distributed training technology, and in particular to a collection communication method, device, chip product, equipment and storage medium. Background Technology

[0002] Distributed training is a method for training machine learning models that distributes the training process across multiple computing nodes in a distributed system to achieve parallel training and improve training efficiency. Each computing node can run one or more processes, and each process can manage one or more GPUs (Graphics Processing Units) to execute distributed training tasks. During the execution of distributed training tasks, processes can communicate using Collective Communications, a communication pattern in distributed systems used for data synchronization between multiple processes (i.e., computing nodes), such as model parameters and gradients, ensuring that all processes (i.e., computing nodes) function as a whole to accelerate the training of the machine learning model.

[0003] In related technologies, before officially starting distributed training, the distributed system divides all processes participating in the distributed training task into multiple communication process groups according to a parallel strategy. For example, for a data parallel (DP) strategy, multiple computing nodes are divided into t computing node groups, where t is an integer greater than 1. The processes corresponding to each of the t computing node groups execute the distributed training task in parallel. Related technologies construct communication process groups across computing node groups. Each communication process group consists of t processes, each from one of the t computing node groups. The t processes in each communication process group communicate with each other using a set communication method to achieve data synchronization among the t processes.

[0004] However, during the execution of a distributed training task, since the members in the communication process group remain unchanged, if a process fails and is unable to participate in the collective communication, the other processes in its communication process group will be affected by the failure of that process, resulting in the failure of the entire collective communication and the termination of the distributed training task. This is not conducive to improving the reliability of the distributed system. Summary of the Invention

[0005] This application provides a collection of communication methods, apparatus, chip products, devices, and storage media. The technical solutions provided by this application include the following.

[0006] According to one aspect of the embodiments of this application, a collective communication method is provided, the method being executed by an adaptation layer located between a training process and an actual communication interface, the training process being used to perform distributed training tasks, and the actual communication interface being used to perform collective communication operations; the method includes: Obtain the liveness status of multiple training processes participating in the ensemble communication; If a faulty process is determined based on the survival status, a working communication process group is reconstructed based on the currently surviving training process, and the working communication process group is used to replace the historical communication process group used before the fault. Call the actual communication interface corresponding to the working communication process group to execute the set communication operation and obtain the set communication result; The results of the aggregated communication are sent to the training process in the working communication process group.

[0007] According to one aspect of the embodiments of this application, a collective communication device is provided, the device being located between a training process and an actual communication interface, the training process being used to perform distributed training tasks, and the actual communication interface being used to perform collective communication operations; the device includes: The survival status acquisition module is used to acquire the survival status of multiple training processes participating in the ensemble communication. The communication group reconstruction module is used to reconstruct a working communication process group based on the currently surviving training process when it is determined that there is a faulty process according to the survival status. The working communication process group is used to replace the historical communication process group used before the fault. The communication result acquisition module is used to call the actual communication interface corresponding to the working communication process group, execute the collection communication operation, and obtain the collection communication result; The communication result sending module is used to send the set communication results to the training process in the working communication process group.

[0008] According to one aspect of the embodiments of this application, a chip product is provided, wherein a computer program is stored in the chip product, the computer program being loaded and executed by a processor to implement the above-described collection communication method.

[0009] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described collection communication method.

[0010] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, the computer program being loaded and executed by a processor to implement the above-described collection communication method.

[0011] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the above-described collection communication method.

[0012] The technical solutions provided in the embodiments of this application may include the following beneficial effects.

[0013] By deploying an adaptation layer between the training process and the actual communication interface, the adaptation layer reconstructs the working communication process group based on the survival status of the training processes participating in the collective communication, and calls the actual communication interface of the working communication process group to obtain the collective communication result, thus realizing collective communication. This makes the training process unaware of changes to the working communication process group and the actual communication interface, thereby making the training process unaware of the fault-tolerant process of collective communication.

[0014] Furthermore, by reconstructing the working communication process group based on the survival status of the training processes participating in the ensemble communication, surviving training processes can continue to participate in the ensemble communication, thus avoiding interruptions and ensuring that the distributed training task is not interrupted. This is beneficial for improving the reliability and fault tolerance of the distributed training system corresponding to the distributed training task. Moreover, by eliminating the need to wait for fault recovery before resuming ensemble communication, the latency introduced during fault recovery can be avoided, thereby helping to maintain the training efficiency of the distributed training system. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of a distributed training system provided in one possible implementation of this application; Figure 2 This is a schematic diagram illustrating data parallelism provided in one possible implementation of this application; Figure 3 This is a schematic diagram of pipeline parallelism provided in one possible implementation of this application; Figure 4 This is a schematic diagram of tensor parallelism provided in one possible implementation of this application; Figure 5 This is a schematic diagram of hybrid parallelism provided in one possible implementation of this application; Figure 6 This is a schematic diagram of a collection communication interruption provided in one possible implementation of this application; Figure 7 This is a schematic diagram of a collection communication interruption provided in another possible implementation of this application; Figure 8 This is a schematic diagram of the adaptation layer provided in one possible implementation of this application; Figure 9 This is a schematic diagram illustrating uninterrupted collection communication provided in one possible implementation of this application; Figure 10 This is a flowchart of a collection communication method provided in one possible implementation of this application; Figure 11 This is a schematic diagram of a communication process group provided in one possible implementation of this application; Figure 12 This is a schematic diagram illustrating uninterrupted collection communication provided in another possible implementation of this application; Figure 13 This is a schematic diagram of a method for obtaining collection communication results provided in one possible implementation of this application; Figure 14 This is a schematic diagram of the distribution set communication results provided in one possible implementation of this application; Figure 15 This is a schematic diagram of the first mapping data provided in one possible implementation of this application; Figure 16 This is a flowchart of a collection communication method provided in another possible implementation of this application; Figure 17 This is a block diagram of a collection communication device provided in one possible implementation of this application; Figure 18 This is a simplified structural block diagram of a computer device provided in one possible implementation of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0018] Please refer to Figure 1 This illustrates a schematic diagram of a distributed training system provided in one possible implementation of this application.

[0019] The distributed training system 100 can refer to a distributed system used for the distributed training of machine learning models. A machine learning model is an algorithm that automatically learns and generates prediction results based on given input data (also called features). For example, a machine learning model can be a neural network model built upon a neural network. Distributed training is a method for training machine learning models by distributing the training process across multiple computing nodes in a distributed system to achieve parallel training and improve the training efficiency of the machine learning model. A distributed system is a system composed of a group of computing nodes that communicate through a network and coordinate their work to complete a common task. In the embodiments of this application, the task used for the distributed training of the machine learning model can be referred to as a distributed training task.

[0020] In one example, the distributed training system 100 may include multiple computing nodes 101. Each computing node 101 can be used to independently process data to provide computing resources. Multiple computing nodes 101 can cooperate to complete the execution of distributed training tasks, such as multiple computing nodes 101 running in parallel.

[0021] Computing node 101 may be implemented as a computer device, such as a server. Exemplarily, computing node 101 may be implemented as at least one of the following: a server equipped with a Central Processing Unit (CPU), a GPU server equipped with a Graphics Processing Unit (GPU), a server equipped with a General-Purpose Computing on Graphics Processing Units (GPGPU), a server equipped with a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a server equipped with a Tensor Processing Unit (TPU), a Field-Programmable Gate Array (FPGA), a server equipped with Tensor Cores, or a server equipped with a Neural Processing Unit (NPU).

[0022] Optionally, the multiple computing nodes 101 can communicate with each other. For example, a communication link is established between the computing nodes 101, enabling them to communicate with each other. This communication link can be implemented through a network, such as a wired network or a wireless network.

[0023] For example, the network architecture of the distributed training system 100 is a leaf-spine network architecture, which includes leaf switches 102 and spine switches 103. Leaf switches 102 can be implemented as Layer 2 switches for connecting computing nodes 101. Spine switches 103 can be implemented as Layer 3 switches responsible for routing and forwarding, and are connected to leaf switches 102. Some computing nodes 101 can form communication links only through leaf switches 102, while others can form communication links simultaneously through leaf switches 102 and spine switches 103. For example, computing nodes H0 and H1 can form a communication link only through leaf switch 0, while computing nodes H0 and H6 can form a communication link through leaf switch 0, spine switch 103, and leaf switch N. This application embodiment does not limit the value of N, which can be set and adjusted according to actual usage requirements. This application embodiment also does not limit the number of switch layers; for example, the distributed training system 100 may also include switches with three or more layers.

[0024] In one example, the distributed training system 100 also includes a master computing node ( Figure 1 (Not shown in the diagram), denoted as the Training Master, is responsible for coordinating and scheduling computing nodes 101 and distributed training tasks, enabling multiple computing nodes 101 to collaboratively execute distributed training tasks. The master computing node centrally manages and allocates tasks, ensuring that all computing nodes 101 work collaboratively to optimize training efficiency. A computing node 101 can be randomly selected from the distributed training system 100 as the master computing node, or a separate master computing node can be set up outside the distributed training system 100; this embodiment does not limit this approach.

[0025] Optionally, the master compute node can also be responsible for the monitoring and control of distributed training tasks, the partitioning of sample data for machine learning models, the status monitoring and control of compute node 101, network configuration management, and fault log analysis. In a feasible example, the master compute node can be implemented as a functional module and deployed on compute node 101.

[0026] In one example, each computing node 101 may be equipped with at least one computing device, which includes at least one of the following: CPU, GPU, GPGPU, DSP, ASIC, TPU, FPGA, Tensor Core, and NPU. For example, if computing node 101 is implemented as a GPU server, multiple GPUs may be configured on computing node 101, and each GPU may be implemented as a GPU card (i.e., a chip), which is the computing device. Each computing node 101 may run a process to utilize at least one computing device on computing node 101 to accelerate computing tasks. In the embodiments of this application, the process used to execute distributed training tasks can be called a training process. The training process is the core execution unit that carries out distributed training tasks. It is responsible for independently completing a series of key operations on a single computing node 101, from data loading, forward propagation, back propagation to gradient calculation, and cooperating with other training processes to jointly complete the distributed training task of the entire machine learning model.

[0027] For example, each computing node 101 may run one or more training processes, and each training process may manage one or more computing devices. For instance, taking a computing node 101 as a GPU server, the GPU server may include multiple GPUs (i.e., multiple graphics cards or GPU cards), and each GPU server may run multiple training processes. Each training process may manage at least one GPU to perform distributed training tasks. This embodiment of the application illustrates an example where each training process corresponds to one GPU.

[0028] In one example, distributed training of a machine learning model is an iterative process, with each iteration corresponding to a distributed training task of one step. The distributed training system 100 completes the iterative training of the machine learning model by sequentially executing the distributed training tasks of each step. Distributed training typically requires multiple parallel techniques to improve training efficiency. Based on the different degrees of parallelism of these techniques, the distributed training task can be divided into subtasks with varying degrees of parallelism.

[0029] For example, the parallel techniques used in distributed training may include at least one of the following: Data Parallel (DP) strategy: Multiple computing nodes (101) are divided into multiple computing node groups, and the sample data used to train the machine learning model is split into multiple data subsets. Different data subsets are assigned to different computing node groups, and each computing node group has the same complete model parameters. Each computing node group consists of the same number of computing nodes, and each computing node group includes at least one computing node. For example, refer to... Figure 2The sample data 201 was divided into two subsets, each processed by a separate set of compute nodes to achieve distributed training of the machine learning model. Compute node set 1 and compute node set 2 executed the distributed training task in parallel.

[0030] In a feasible example, a data parallelism strategy can also be adopted with computing devices as the smallest unit, such as dividing the computing devices into multiple computing device groups, and having multiple computing device groups execute distributed training tasks in parallel. In this case, the computing device group is equivalent to a computing node group. This application embodiment uses a computing node group as an example for illustration.

[0031] Pipeline Parallel (PP) strategy: This involves partitioning the machine learning model across layers, distributing the data (such as model parameters) of one model layer across a subset of computing nodes, and distributing the data of another model layer across a separate subset of computing nodes. The model layers are the foundational data structures of the machine learning model, and different model layers can perform different data processing operations, such as convolution, pooling, normalization, linear transformation, and logistic regression. For example, refer to... Figure 3 The model parameters of a portion of the model layers of the machine learning model are assigned to computing node 302, while the model parameters of the remaining model layers are assigned to computing node 303. Computing nodes 302 and 303 complete the distributed training of the machine learning model based on sample data 301.

[0032] Tensor Parallel (TP) strategy: This involves partitioning the machine learning model within each layer, that is, splitting the data (such as model parameters) of the same model layer across different computation nodes. For example, refer to... Figure 4 For each model layer of the machine learning model, the model parameters of the model layer are evenly distributed across computing nodes 402 and 403. Computing nodes 402 and 403 are simultaneously trained based on sample data 401. PP and TP represent model parallelism.

[0033] Hybrid Parallel (3D Parallel) Strategy: Simultaneously applying dynamic programming (DP), power processing (PP), and total response (TP). For example, refer to... Figure 5Taking DP=8, PP=12, and TP=4 as an example, the sample data 501 is split into 8 data subsets according to the parallelism of DP. Each data subset is used to train a complete machine learning model. That is, the multiple computing nodes 101 in the distributed training system 100 are divided into 8 computing node groups, and each computing node group is used to process a data subset. For any complete machine learning model, it is split according to the parallelism of PP, so that different model layers are assigned to different computing nodes 101. For example, taking computing node group 502 as an example, computing node group 502 includes 12 computing device groups. The computing devices (such as GPUs) in each computing device group can be on the same computing node 101 or on different computing nodes 101. The model parameters of the model layers of the machine learning model can be distributed proportionally to the 12 computing device groups.

[0034] For each model layer, it can be further partitioned according to the parallelism of TP, so that different model parameters are assigned to different computing devices (such as GPUs) within the same computing device group. For example, for any computing device group in 12 computing device groups, the model parameters of the corresponding model layer can be divided into 4 parts, and each part can be assigned to one computing device in that computing device group. The values ​​in the above embodiments are merely exemplary and illustrative. This application does not limit the number of computing node groups, the number of computing nodes in computing node groups, the number of computing device groups, or the number of computing devices in computing device groups.

[0035] The embodiments of this application do not limit the smallest unit of the parallel strategy, such as computing devices (corresponding to training processes) or computing nodes.

[0036] In one example, during the distributed training of a machine learning model using a data-parallel or hybrid parallel strategy, after each training cycle, each computing node group needs to exchange training state data (such as model parameters, parameter gradients, activation values, etc.) and update the overall model parameters of the machine learning model to complete the entire distributed training task (implemented by the corresponding training process). For instance, before the next training cycle begins, the training state of each computing node group needs to be synchronized to ensure consistency, thereby achieving distributed training of the machine learning model. For example, regarding model parameters, before the next training cycle begins, the updated overall model parameters of the machine learning model will be synchronized to each computing node group to ensure consistency, thus achieving parallel training of the machine learning model parameters.

[0037] The exchange of training state data can be achieved by the training processes corresponding to different computing node groups. Optionally, collective communications can be used to synchronize training state data between each computing node group. Collective communications is a communication mode in distributed systems that can be used for data synchronization between multiple computing nodes (i.e., training processes).

[0038] For example, for data parallelism or hybrid parallelism, the distributed training system 100 can use aggregated communication to facilitate communication between training processes, thereby synchronizing training state data among the various computing nodes 101 in the distributed training system 100. This ensures that all computing nodes 101 (training processes) function as a whole to accelerate the training of machine learning models. In other words, the technical solutions provided in this application are applicable to any distributed training system employing a data parallelism strategy or a hybrid parallelism strategy.

[0039] The training processes running on compute node 101 are logical entities participating in ensemble communication. For example, within a communication domain (i.e., a communication process group), each participating training process has a unique rank number. If eight training processes participate in ensemble communication, each process will be assigned one of eight rank numbers from 0 to 7. If a computing device (such as a GPU) is used for ensemble communication, a training process is typically bound to one computing device. If each compute node 101 has multiple computing devices (e.g., eight computing devices), then during distributed training, each compute node 101 will typically have a corresponding number of training processes (e.g., eight), meaning each compute node 101 can correspond to eight different rank numbers. Training processes participating in communication can communicate across compute node groups. Training processes across compute node groups can communicate via a network (or any supported connection method). Training processes within the same compute node 101 can communicate via any supported connection method, such as PCIe (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard), private connections, etc. This embodiment does not limit the specific methods used.

[0040] In related technologies, ensemble communication requires initializing the communication process group at the start of distributed training according to different parallel strategies. The members of the communication process group remain unchanged during training. A communication process group is a set of training processes from different computing node groups. During distributed training, the training processes within a communication process group communicate with each other but not with training processes outside that group. If the parallel strategy and parallelism parameters need to be adjusted, the communication process group needs to be reinitialized for ensemble communication. Furthermore, if a training process fails and cannot participate in communication during ensemble communication, the entire ensemble communication is interrupted, leading to the interruption of the entire distributed training task. The performance of ensemble communication directly affects the speed of the distributed training task, determining whether all computing nodes (training processes) in the cluster can work together to accelerate model training. For example, if one training process fails in ensemble communication, the other training processes in the communication process group are in a waiting state, causing ensemble communication to be interrupted, and consequently, the distributed training to be interrupted.

[0041] In one example, we take the Ring-AllReduce (Ring-AllReduce) aggregate communication method as an example. Ring-AllReduce forms a ring topology for the training processes in the same communication process group. Each training process only exchanges data with its two adjacent training processes. The data of each training process is passed sequentially according to the ring topology and completes AllReduce calculations (such as summation, maximum value, average, etc.).

[0042] When using Ring-AllReduce, the communication topology of the communication process group is a ring topology. When the number of training processes in the communication process group is m, the m-th training process and the first training process have a communication relationship, and the i-th training process and the (i-1)-th and (i+1)-th training processes have a communication relationship, where i is an integer greater than 1 and less than m.

[0043] like Figure 6 As shown, the communication process group 600 includes 4 training processes. The communication topology of the communication process group 600 is a ring topology structure. The 4 training processes are connected end to end in sequence. The previous training process only transmits data to the next training process. For example, training process 0 only transmits data to training process 1, training process 1 only transmits data to training process 2, training process 2 only transmits data to training process 3, and training process 3 only transmits data to training process 0.

[0044] If training process 3 fails, the communication topology is interrupted by training process 3, and training processes 0, 1, and 2 all enter a waiting state, resulting in a complete communication breakdown for the entire set. Furthermore, since the next communication target for each training process is fixed when the communication topology is created, training process 2 cannot bypass training process 3 to transfer data if training process 3 fails.

[0045] like Figure 7 As shown, when using data parallelism or hybrid parallelism, different training processes in communication process group 600 need to calculate their gradients independently and then perform gradient averaging. In step 2, if training process 3 fails (e.g., the computing device or node where training process 3 resides fails), the aggregate communication corresponding to communication process group 600 is interrupted, leading to the interruption of distributed training. After training process 3 recovers from the failure, the distributed training system can restart distributed training for step 2.

[0046] This application provides a method for addressing the problem of ensemble communication interruption caused by training process failures. The method deploys an adaptation layer between the training process and the actual communication interface. This layer dynamically updates the members of the working communication process group based on the survival status of the training processes participating in ensemble communication, thereby reconstructing the working communication process group. In the event of a failed process, the currently surviving training processes can continue to perform ensemble communication operations, thus avoiding ensemble communication interruption and ensuring that the distributed training task is not interrupted. This improves the reliability and fault tolerance of the distributed training system. Furthermore, by eliminating the need to wait for fault recovery before resuming ensemble communication, the delay introduced during fault recovery is avoided, which helps maintain the training efficiency of the distributed training system.

[0047] In addition, since the training process only interacts with the adaptation layer, and the adaptation layer reconstructs the working communication process group and calls the actual communication interface of the working communication process group to obtain the collection communication interface based on the survival status of the training process participating in the collection communication, the training process is unaware of the changes to the working communication process group and the actual communication interface, thus achieving the training process being unaware of the fault tolerance process of the collection communication.

[0048] In one example, the aforementioned adaptation layer sits between the training process and the actual communication interface. The training process executes distributed training tasks, while the actual communication interface performs aggregated communication operations. The actual communication interface is the aggregated communication interface (e.g., the communication process group interface) corresponding to the working communication process group. The training process can call the aggregated communication interface corresponding to the adaptation layer to initiate an aggregated communication request. The adaptation layer, in response to the aggregated communication request, can call the actual communication interface corresponding to the working communication process group to perform the actual aggregated communication operation for each training process within the working communication process group. For example, when the training process calls the aggregated communication interface of the adaptation layer, it can send the identifier information of the proxy communication process group, the training state data to be synchronized, etc., to the aggregated communication interface of the adaptation layer (e.g., the communication process group interface). The adaptation layer processes the training state data to be synchronized by calling the actual communication interface of the working communication process group to obtain the aggregated communication result.

[0049] The Communication Process Group Interface (CPM) is used for collective communication within a group of communication processes. For example, the CPM is an abstraction layer and programming framework for managing communication process groups, encapsulating and scheduling the underlying collective communication interface. It can be used to organize training processes and manage the communication context, defining how a group of training processes can be organized into a cooperating unit.

[0050] The communication process group interface establishes a background communication thread for each communication process group. The training thread (computation thread) is responsible for forward and backward propagation computation, while the communication thread handles the collection communication task in the background. The two collaborate asynchronously through a queue, thus achieving overlap between computation and communication. The Collective Communication Interface is a standardized programming interface that provides functional primitives for data exchange (i.e., collection communication functions). It defines the semantics of data exchange operations (i.e., communication algorithms), such as Broadcast, Reduce, Allreduce, and Gather. The Collective Communication Interface can be used to call collection communication functions, thereby enabling data exchange between multiple training processes.

[0051] In this embodiment, the adaptation layer can be implemented as a general-purpose collective communication interface for the training process, which can be deployed on each computing node 101 (or computing device) in the distributed training system 100 for the training process to call to achieve collective communication. The adaptation layer can also be implemented as a functional layer in the above-mentioned general-purpose collective communication interface for dynamically reconstructing the working communication process group. This embodiment does not limit the adaptation layer.

[0052] For example, the adapter layer is implemented as a standard communication process group interface, and can achieve fault tolerance for aggregate communication by creating working communication process groups internally for actual communication. The working communication process group is the process group used to actually perform aggregate communication operations. The adapter layer has a corresponding proxy communication process group, which is a communication process group divided before the start of distributed training. This proxy communication process group remains unchanged during distributed training and is accessible to all training processes. The communication topology of the communication process group can be a ring, tree, or other topological structure to determine the communication methods and paths between training processes. The communication process group interface can be a wrapper around the aggregate communication backend, such as a wrapper around a low-level dedicated communication library. The aggregate communication functions corresponding to the communication process group interface can be used to perform aggregate communication operations on the communication process group.

[0053] For example, refer to Figure 8 The adapter layer 800 can provide at least one of the following capabilities: (1) Implement a standard, complete communication process group interface, which may include an object construction interface (constructor) and interfaces for communication operators such as Broadcast, Gather, Reduce, Allreduce, Scatter, Allgather, and Reduce Scatter, so as to maintain interface consistency with upper-layer callers (such as training processes).

[0054] From the caller's perspective, the adaptation layer 800 is also a communication process group interface. The parameters created in the communication process group interface are: size (i.e., world size): the number of members in the proxy communication process group participating in the collective communication; rank: the unique identifier of the training process in the proxy communication process group, and the value range of rank is [0, 1, 2, ..., size-1], which is numerically continuous.

[0055] (2) After the proxy communication process group is created, its corresponding parameters remain unchanged for the upper-level caller.

[0056] (3) The adaptation layer 800 dynamically creates and destroys the working communication process group and the actual communication interface 801 of the working communication process group. The actual communication interface 801 is a standard and complete communication process group interface built for the working communication process group.

[0057] (4) The adaptation layer 800 dynamically executes the collection communication operation by calling the actual communication interface 801 of the working communication process group to obtain the collection communication result.

[0058] The working communication process group (i.e., the actual communication interface 801) may belong to the same operating system process as the proxy communication process group (i.e., the communication process group interface or the adaptation layer 800), or may belong to two independent operating system processes as needed. This application embodiment does not limit this.

[0059] like Figure 9 As shown, when communication process group 600 is a proxy communication process group, in the event of a failure in training process 3, the adaptation layer reconstructs the working communication process group and its actual communication interface based on training processes 0, 1, and 2, while keeping communication process group 600 unchanged. The adaptation layer performs a collective communication operation on each training process within the working communication process group by calling the corresponding actual communication interface, obtaining the collective communication result. This ensures that collective communication can still be achieved between training processes 0, 1, and 2, thus preventing interruption of collective communication and consequently ensuring uninterrupted distributed training.

[0060] In this embodiment of the application, the distributed training system 100 may further include a central controller ( Figure 1 (Not shown in the diagram). For example, the central controller can be used to detect the liveness status of the training process, and the aforementioned adaptation layer can learn about the liveness status of the training process from the central controller.

[0061] Optionally, a central controller can be deployed for each communication process group, or a single central controller can be deployed for all communication process groups. The central controller can be implemented as a functional layer and deployed in the computing node 101 of the distributed training system 100, or in the main computing node, or in a computing node located outside the distributed training system 100. This application embodiment does not limit this.

[0062] In one example, a single distributed training system 100 can be implemented as a single training replica. If multiple training replicas are built simultaneously and training state data (such as gradients) is synchronized among the multiple training replicas, multi-replica training can be achieved. The technical solutions provided in the embodiments of this application are applicable to any of the multiple training replicas.

[0063] The following will use method embodiments to describe in detail the technical solutions provided in the embodiments of this application. For content not described in the method embodiments, please refer to the above embodiments, which will not be repeated here.

[0064] Please refer to Figure 10 It illustrates a flowchart of a collection communication method provided in one possible implementation of this application, wherein the execution entity of each step of the method can be... Figure 1The distributed training system 100 shown includes a computing node 101, such as an adaptation layer within the computing node 101. The method may include at least one of the following steps (1001-1004).

[0065] Step 1001: Obtain the survival status of multiple training processes participating in the set communication.

[0066] In this embodiment, the distributed training system includes multiple computing nodes, and training processes on the multiple computing nodes are used to execute distributed training tasks. Exemplarily, the distributed training system can employ a data parallel strategy to execute distributed training tasks, or it can employ a hybrid parallel strategy; this embodiment does not limit this approach. For example, the training process on each computing node is used to train a complete machine learning model, and the training processes on multiple computing nodes can achieve distributed training of the machine learning model.

[0067] Before starting the distributed training task, the distributed training system can divide the training processes on multiple computing nodes into at least one communication process group according to a parallel strategy. The training processes in each communication process group can communicate with each other collectively. This at least one communication process group can serve as a proxy communication process group for the adaptation layer, and the proxy communication process group remains unchanged during the distributed training process.

[0068] For example, in a data parallel strategy, multiple computing nodes in a distributed training system are divided into multiple computing node groups. These groups are used to execute distributed training tasks in parallel. The training processes running on the multiple computing nodes are initialized into at least one communication process group. Each training process in a communication process group comes from one of the multiple computing node groups. Each computing node group may include the same number of computing nodes, and each communication process group may include the same number of training processes.

[0069] For example, refer to Figure 11 For the hybrid parallel strategy, the distributed training system divides the computing nodes into n computing node groups. Each computing node group includes K+1 computing nodes 1101. Each computing node 1101 has multiple computing devices 1102 (such as GPU cards). Each computing device 1102 runs a training process, that is, each training process manages one computing device 1102.

[0070] Since there are n computing node groups globally, each communication process group includes n training processes. For any training process in a communication process group, its position within its own computing node group is the same as the positions of other training processes in that communication process group within the computing node groups of other training processes. For example, the first training process on the Kth computer node in each computing node group can form a communication process group. The total number of communication process groups is TP. PP and TP are the number of computing nodes in each computing node group, and PP is the number of computing devices 1102 (i.e., training processes) in each computing node 1101.

[0071] Optionally, the adaptation layer executes step 1001 upon receiving a collective communication request from any of the training processes in a plurality of training processes. That is, the adaptation layer may execute step 1001 before calling the actual communication interface of the historical communication process group. The adaptation layer may only begin executing the collective communication operation upon receiving collective communication requests from each training process in a communication process group (such as a working communication process group). For example, the adaptation layer may only begin executing the collective communication operation upon receiving collective communication requests from all currently active training processes in the communication process group. Here, the collective communication request is a request used to trigger collective communication, and the collective communication request may include at least one of the following: the identification information of the proxy communication process group to which the training process belongs, the identification information of the communication operator, the training state data to be synchronized, and the identification information of the training process.

[0072] The aforementioned training processes may belong to the same agent communication process group corresponding to the distributed training system. This application embodiment uses the collective communication of a certain agent communication process group for illustration.

[0073] The survival status of a training process indicates whether it is inactive and identifies inactive processes as faulty processes. For example, a training process can be identified as active if it can run normally, and as inactive if it cannot. This includes situations such as the training process freezing, exiting, crashing, or encountering an error, as well as failures in the associated computing device or node. The adaptation layer can obtain the survival status of multiple training processes from the central controller.

[0074] Collective communication is a synchronization mode in distributed training systems. For example, during distributed training, collective communication can be used to synchronize gradients among the various computing nodes (training processes) in the system, thereby enabling global updates of the machine learning model. In the initial stage of distributed training, collective communication can be used to synchronize model parameters among the various computing nodes (training processes) to initialize the distributed training system. The collective communication operation is a process that implements collective communication for training processes within a communication process group.

[0075] In this embodiment, although the adaptation layer declares a proxy communication process group to external entities (such as the training process), it actually uses a working communication process group for aggregated communication, so that the currently surviving training processes are unaware of changes in aggregated communication. This embodiment does not limit the method of aggregated communication described above; it can be set and adjusted according to actual usage requirements.

[0076] Step 1002: If a faulty process is determined based on the survival status, a working communication process group is reconstructed based on the currently surviving training process. The working communication process group is used to replace the historical communication process group used before the fault.

[0077] A faulty process is a training process that has failed. The adaptation layer can identify inactive training processes as faulty processes and currently surviving training processes as healthy processes. Faulty processes cannot continue to participate in aggregate communication, while healthy processes can. The working communication process group is the group of communication processes actually used to perform aggregate communication. In other words, the adaptation layer actually performs aggregate communication operations on each training process in the working communication process group.

[0078] Distributed training of machine learning models is an iterative process. Before a faulty process is detected, the adaptation layer can directly identify the proxy communication process group as the working communication process group. Upon detecting a faulty process, the adaptation layer reorganizes the currently surviving training processes into the working communication process group. For the current iteration, the working communication process group used by the previous iteration can be referred to as the historical communication process group.

[0079] In one example, the adaptation layer may update only the communication relationships of the faulty processes within the communication topology of the historical communication process group to obtain the updated communication topology, thereby reconstructing the working communication process group. For example, the reconstruction process of the working communication process group may include the following:

[0080] S11. Based on the communication topology corresponding to the historical communication process group, determine at least one associated process that has a communication relationship with the faulty process. The communication topology is used to indicate the communication relationship of each process in the historical communication process group in the collective communication.

[0081] Communication relationships can be used to indicate data transfer relationships between training processes. For example, if a communication relationship exists between a first training process and a second training process, the first training process is used to transfer data to the second training process. The communication topology can be a topology formed by communication relationships between multiple training processes. Optionally, the communication topology of a communication process group can be implemented as a ring structure, tree structure, star structure, mesh structure, or any other topology, which is not limited in this embodiment. Specifically, for a tree structure, training processes are connected hierarchically, forming a structure similar to an inverted tree; for a star structure, all training processes are connected to the same training process; for a mesh structure, training processes are connected via multiple paths, forming a structure similar to a fishing net. For ease of explanation and understanding, this embodiment uses a ring structure as an example.

[0082] For example, the adaptation layer can determine the communication topology of a communication process group (such as a historical communication process group) based on the parallel strategy and aggregate communication method of the distributed training system. For instance, if the parallel strategy of the distributed training system is data parallelism, communication relationships can be established between multiple training processes in the communication process group based on a ring structure, generating the communication topology of the communication process group. The adaptation layer can also obtain the communication topology of the communication process group from the aggregate communication library of the distributed training system; this embodiment does not limit this. The aggregate communication library refers to a library used to implement communication between multiple training processes in distributed training.

[0083] Optionally, in the initial stage of distributed training, the adaptation layer automatically acquires the communication topology of the communication process group. During distributed training, the adaptation layer synchronizes data according to the communication topology of the communication process group to jointly complete the collective communication. Figure 12As shown, the historical communication process group 1200 includes four training processes: rank0 to rank3. For each stage of the distributed training task, after the four training processes have completed their respective gradient calculations, the gradients are passed according to the communication topology of the historical communication process group 1200 until each training process obtains the global gradient (which can be the average of all gradients). Each training process updates its own machine learning model parameters based on the global gradient, thereby completing the distributed training of the machine learning model. The communication topology of the historical communication process group 1200 is a ring structure, which can be used to indicate communication relationships between training process 0 (rank0) and training process 1 (rank1), training process 1 (rank1) and training process 2 (rank2), training process 2 (rank2) and training process 3 (rank3), and training process 3 (rank3) and training process 0 (rank0). The adaptation layer sequentially passes gradients among the four training processes according to the communication topology until each training process obtains the global gradient.

[0084] Optionally, the adaptation layer can obtain different numbers of associated processes or different associated processes depending on the communication topology of different structures. For example, for many-to-many aggregate communication, all training processes except the faulty process can be identified as associated processes; for many-to-one aggregate communication, the root process (on which all data is aggregated) can be identified as the associated process of the faulty process; for ring-structured aggregate communication, the previous training process and the next training process of the faulty process can be identified as the associated processes of the faulty process. This application embodiment does not limit this.

[0085] For example, refer to Figure 12 Taking a ring-shaped communication topology as an example, if training process 2 (i.e., rank2) is a faulty process, the adaptation layer can determine training process 1 (i.e., rank1) and training process 3 (i.e., rank3) as associated processes of training process 2 (i.e., rank2).

[0086] S12, for the communication topology, clear the communication relationship between the faulty process and at least one associated process, and adjust the communication relationship between the remaining processes to obtain the updated communication topology.

[0087] The adaptation layer only updates the communication relationship between the faulty process and at least one associated process, and maintains the communication relationship between other training processes (including the remaining processes and associated processes) to reduce the workload of adjusting the communication topology, thereby reducing the workload of rebuilding the working communication process group.

[0088] Optionally, the adaptation layer can directly remove the faulty process from the communication topology, adjust the communication relationship between at least one associated process, and the communication relationship between at least one associated process and the remaining processes, to obtain an updated communication topology. Here, the remaining processes refer to the training processes in the historical communication process group, excluding the faulty process and at least one associated process.

[0089] For example, when the communication topology of the historical communication process group is a ring structure, if the number of processes in the historical communication process group is m, then the m-th training process in the historical communication process group has a communication relationship with the first training process, and the i-th training process in the historical communication process group has a communication relationship with the (i-1)-th training process and the (i+1)-th training process, respectively, where i is an integer greater than 1 and less than m.

[0090] If the i-th training process is a faulty process, for the communication topology, the communication relationships between the i-th training process and the (i-1)-th and (i+1)-th training processes are cleared, and a communication relationship is established between the (i-1)-th and (i+1)-th training processes to obtain the updated communication topology.

[0091] In the case that the first training process is a faulty process, for the communication topology, the adaptation layer clears the communication relationship between the first training process and the second training process and the m-th training process, and establishes a communication relationship between the m-th training process and the second training process, thus obtaining the updated communication topology.

[0092] For example, refer to Figure 12 In the event of a failure in training process 2, the adaptation layer clears the communication relationships between training process 2 and training processes 1 and 3 respectively, and establishes a communication relationship between training process 1 and training process 3, resulting in an updated communication topology. The communication relationships between training process 0 and training process 1 remain unchanged, as do the communication relationships between training process 3 and training process 0.

[0093] In a feasible example, the working communication process group is reconstructed by the central controller in the distributed training system. The method by which the central controller reconstructs the working communication process group is the same as the method by which the adaptation layer reconstructs the working communication process group. After reconstructing the working communication process group, the central controller provides the working communication process group to the adaptation layer.

[0094] This embodiment ensures uninterrupted aggregate communication and avoids introducing delays in fault recovery by removing faulty processes from the communication topology and establishing communication relationships between the remaining processes. This improves the performance of aggregate communication and, consequently, the performance of the distributed training system.

[0095] Furthermore, by making local adjustments to communication relationships based on the communication topology, without making global adjustments, the communication topology can be updated quickly. This helps improve the efficiency of communication topology updates and thus reduces the impact on aggregated communication.

[0096] S13, create a working communication process group based on the updated communication topology.

[0097] The updated communication topology is determined as the communication topology of the working communication process group, and the training processes in the updated communication topology, i.e., all currently alive training processes in the historical communication process group, are combined into the working communication process group.

[0098] Optionally, if it is determined from the survival status that there are no faulty processes, the historical communication process group is identified as the working communication process group.

[0099] S14, Based on the working communication process group, create the actual communication interface for the working communication process group.

[0100] Optionally, the adaptation layer constructs a new set communication interface for the working communication process group using a standard set communication interface construction method, and determines the new set communication interface as the actual communication interface of the working communication process group.

[0101] For example, refer to Figure 8 If the actual communication interface 801 is the actual communication interface of the historical communication process group, the adaptation layer will reconstruct the working communication process group, which includes M training processes, and the actual communication interface 802 of the working communication process group in the event of a failed process in the historical communication process group. The historical communication process group includes N training processes, where M is less than N.

[0102] Optionally, after constructing the actual communication interface 802, the adapter layer can destroy the actual communication interface 801.

[0103] In this embodiment of the application, if the training process fails during a certain iteration, the adaptation layer will no longer perform the set communication operation for the training process. Instead, after the training process is restored or replaced, the set communication operation will be performed for the training process in subsequent stages, thereby achieving uninterrupted set communication and thus uninterrupted distributed training.

[0104] Alternatively, you can first try to recover the faulty process. If the faulty process cannot be recovered, then remove the faulty process directly from the communication topology.

[0105] This embodiment reconstructs the working communication process group based on the communication topology of the historical communication process group, which can ensure that the currently surviving training processes can conduct correct collective communication, thereby improving the performance and stability of the distributed training system.

[0106] Step 1003: Call the actual communication interface corresponding to the working communication process group to perform the set communication operation and obtain the set communication result.

[0107] Upon receiving a collective communication request from each training process in the working communication process group, the adaptation layer calls the actual communication interface corresponding to the working communication process group. Through the collective communication function (such as a communication operator) corresponding to the actual communication interface, it performs collective communication operations on each training process in the working communication process group, thereby achieving data synchronization between the training processes in the working communication process group.

[0108] For example, refer to Figure 12 After rebuilding the working communication process group, the adaptation layer calls the actual communication interface of the rebuilt working communication process to perform a set communication operation for rank0, rank1, and rank3 according to the communication topology of the working communication process, thereby achieving data synchronization between rank0, rank1, and rank3. Training process 2 no longer participates in the set communication to avoid interrupting it. Training process 2 also no longer participates in executing distributed training tasks.

[0109] Optionally, the adaptation layer can determine the aggregated communication result returned by the actual communication interface as the final aggregated communication result, or it can calibrate the aggregated communication result returned by the actual communication interface for faulty processes to obtain the final aggregated communication result. This embodiment of the application does not limit this. For example, if there is no faulty process in the proxy communication process group, the aggregated communication result returned by the actual communication interface can be determined as the final aggregated communication result; if there is a faulty process in the proxy communication process group, the calibrated aggregated communication result can be determined as the final aggregated communication result.

[0110] In one example, when rebuilding a working communication process group, the process of obtaining the collection communication results may include the following.

[0111] S21, call the actual communication interface, perform the set communication operation, and obtain the initial set communication result. The initial set communication result is the result obtained by performing the set communication operation on the training process in the working communication process group.

[0112] The initial set communication result can be obtained by processing the data provided by each training process in the working communication process group. For example, based on the training state data provided by each training process in the working communication process group, the communication operator corresponding to the actual communication interface is executed to obtain the initial set communication result. The communication operator corresponding to the actual communication interface can be implemented as at least one of the following: Broadcast, Gather, Reduce, Allreduce, Scatter, Allgather, Reduce Scatter.

[0113] For example, taking Reduce as an example, if the operation type of Reduce is SUM (summation operation), the root process in the worker communication process group will receive the data sent by other training processes in the worker communication process group, and combine it with its own data to perform summation and obtain the set communication result.

[0114] For example, refer to Figure 13 The historical communication process group 1301 includes four training processes: rank0 to rank3. The communication operator corresponding to the historical communication process group 1301 is Reduce, and rank0 is the root process. In the absence of faulty processes, the communication operator can aggregate the training state data of the four training processes to rank0 and sum the training state data of the four training processes to obtain the set communication result. For example, the communication operator sums [1], [2], [3] and [4] to obtain the set communication result

[10] .

[0115] If rank2 fails, the historical communication process group 1301 is updated to the working communication process group 1302. The communication operator can aggregate the training state data of rank0, rank1 and rank3 into rank0, and sum the training state data of the three training processes to obtain the initial set communication result. For example, the communication operator sums [1], [2] and [4] to obtain the initial set communication result [7].

[0116] S22, determine the communication operator corresponding to the actual communication interface. The communication operator is used to process the data that each training process in the working communication process group needs to synchronize.

[0117] Optionally, the adaptation layer can determine the actual communication operator used by the actual communication interface to perform the collective communication operation based on the communication operator corresponding to the working communication process group or the communication operator corresponding to the proxy communication process group. The communication operator corresponding to the working communication process group can be set and adjusted according to actual usage requirements, and this embodiment does not limit this. The communication operator corresponding to the working communication process group can be the same as the communication operator corresponding to the proxy communication process group.

[0118] S23, according to the operation type corresponding to the communication operator, perform optimization operation on the initial set communication result to obtain the set communication result.

[0119] The operation type is used to indicate the type of operation performed by the communication operator, such as summation, arithmetic mean, maximum value, minimum value, bitwise AND, bitwise OR, bitwise XAND, product, etc. This application embodiment does not limit this.

[0120] Optionally, each operation type may correspond to an optimization operation. For example, the optimization process for the initial set communication result may also include at least one of the following.

[0121] (1) When the operation type corresponding to the communication operator is the first type, the average operation is performed on the initial set communication result to obtain the basic set communication result; the set communication result is obtained based on the basic set communication result and the number of processes in the historical communication process group. The first type includes: summation operation.

[0122] For example, when the operation type is summation, the set communication result is obtained by multiplying the basic set communication result by the number of processes in the historical communication process group.

[0123] Optionally, based on the operation type, the adaptation layer can also combine user strategies and data (data to be synchronized in the training process) distribution to process the initial set communication results.

[0124] For example, when the operation type is summation, the initial set communication result can be calibrated according to a user-specified strategy or the following optimization operation can be referenced: If the data are independent and identically distributed, the product between the initial set communication result and the scaling factor can be determined as the set communication result. The scaling factor is the ratio between the number of training processes in the agent communication process group and the number of training processes in the worker communication process group.

[0125] (2) When the operation type corresponding to the communication operator is the second type, the initial set communication result is determined as the set communication result. The second type includes at least one of the following: calculating the arithmetic mean, calculating the maximum value, calculating the minimum value, bitwise AND, bitwise OR, and bitwise XAND.

[0126] For example, when the operation type is arithmetic average, since the data is uniformly distributed, the loss of some data will not have a significant impact on the final result. Therefore, optimization operations can be omitted on the initial set communication result, and the initial set communication result can be directly determined as the set communication result. The adaptation layer can also calibrate the initial set communication result according to a user-specified strategy, but this application embodiment does not limit this.

[0127] For example, when the operation type is to find the maximum or minimum value, the adaptation layer may not perform optimization operations on the initial set communication results, or it may calibrate the initial set communication results according to the user-specified strategy. This application embodiment does not limit this.

[0128] For example, when the operation type is bitwise AND or bitwise XAND, the adaptation layer may not perform optimization operations on the initial set communication results, or it may calibrate the initial set communication results according to the user-specified strategy. This application embodiment does not limit this.

[0129] (3) When the operation type corresponding to the communication operator is the third type, the scaling factor is determined according to the number of processes in the working communication process group; the initial set communication result is adjusted according to the scaling factor to obtain the set communication result. The third type includes the product operation.

[0130] The scaling factor is determined by the ratio of the number of training processes in the proxy communication process group to the number of training processes in the worker communication process group. The initial set communication result is then subjected to an exponential operation based on the scaling factor to obtain the final set communication result.

[0131] For example, when the operation type is product operation, if the data to be synchronized is independent and identically distributed, the adaptation layer can calibrate the initial set communication results according to the scaling factor, or it can calibrate the initial set communication results according to the user-specified strategy. This application embodiment does not limit this.

[0132] For example, refer to Figure 13 In the case of rank2 failure, due to the lack of rank2 data (e.g., [3]), the adaptation layer cannot obtain accurate data for the failure-free case (i.e.,

[10] ). For the summation operation, the overall mean can be estimated by the mean of the data, that is, first average the initial set communication results (i.e., [7]), and then scale it according to the original number of training processes (i.e., 4) to obtain the set communication results: [7] (4 / 3)≈9.33]. Here (4 / 3) is the scaling factor. Although the results of the set communication still deviate from the true value

[10] , they are an improvement over the initial set communication results, which can effectively reduce the accuracy loss that may be caused by the set communication fault tolerance process.

[0133] In distributed training systems, since Reduce or Allreduce is usually used to calculate the gradient average between node groups (the operator is AVG or SUM, and when it is SUM, it is usually averaged again afterward), and the lack of gradient data for calculating node groups usually does not significantly affect the overall gradient result after averaging, the set communication result under set communication fault tolerance scenario has a limited impact on the overall training of the distributed training system for Reduce or Allreduce.

[0134] Reduce or Allreduce typically accepts two arguments: input, which can be a tensor used to provide the input data for the training process and write the output back; and output, which can also be a tensor used to store the output of the training process. Optionally, input and output can share a single tensor.

[0135] This embodiment calibrates the initial set communication results corresponding to the working communication process group, thereby reducing the accuracy loss caused by set communication fault tolerance and ensuring the performance of the distributed training system.

[0136] In addition, this embodiment calibrates the initial set communication results by using different optimization operations according to different operation types. This helps to improve the rationality of the calibration of the initial set communication results, thereby improving the accuracy of the set communication results.

[0137] Step 1004: Send the aggregate communication results to the training process in the working communication process group.

[0138] For example, refer to Figure 13 For the communication operator Reduce, the adaptation layer only needs to send the collection communication results to the root process rank0. (See reference) Figure 14 For the communication operator Allreduce, the adaptation layer needs to send the set communication results to the four training processes in the working communication process group 1401 respectively. in0 to in3 are the data to be synchronized corresponding to the four training processes respectively, and Out (i.e. Output above) is the set communication result.

[0139] Each training process in the worker communication process group needs to wait for the aggregate communication to complete to ensure that all training processes have received the required data for the next stage of distributed training.

[0140] In one example, after the adaptation layer reconstructs the working communication process group, it will synchronously construct the identification information of the currently surviving training process in the working communication process group. The adaptation layer can no longer use the identification information of the currently surviving training process in the proxy communication process group to distribute the aggregate communication results. To address this issue, step 1004 may also include the following.

[0141] S31, determine the first mapping data based on the working communication process group and the historical communication process group. The first mapping data is used to indicate the identification information of each training process in the working communication process group in the historical communication process group and the working communication process group, respectively.

[0142] In this embodiment, the historical communication process group specifically refers to the aforementioned proxy communication process group. That is, the first mapping data is used to indicate the correspondence between the identification information of the currently active training process in the working communication process group and its corresponding identification information in the proxy communication process group.

[0143] For any training process in the working communication process group, the identification information corresponding to the training process in the working communication process group can be stored in correspondence with the identification information corresponding to the proxy communication process group to obtain the first mapping data.

[0144] Optionally, the mapping data between the proxy communication process group and the historical communication process group can be adjusted according to the working communication process group to obtain the first mapping data.

[0145] For example, refer to Figure 15 The proxy communication group process group 1501 (a historical communication process group) includes n training processes, whose ranks (i.e., identifiers) are [0, 1, 2, ..., n-1], and size = n. Due to the failure of rank1, the working communication process group 1502 is reconstructed. The working communication process group 1502 includes n-1 training processes, whose ranks are [0, 1, 2, ..., n-2], and size = n-1. The first mapping data can then be used to indicate that rank0 in proxy communication group process group 1501 corresponds to rank0 in working communication process group 1502, rank2 in proxy communication group process group 1501 corresponds to rank1 in working communication process group 1502, rank3 in proxy communication group process group 1501 corresponds to rank3 in working communication process group 1502, and so on.

[0146] S32, for the target training process in the working communication process group, send the set communication results to the target training process according to the first mapping data.

[0147] The target training process can be any training process in the working communication process group. For example, when the communication operator is Reduce, the target training process can be the root process, which can be any training process in the working communication process group; when the communication operator is Allreduce, the target training process can be any training process in the working communication process group.

[0148] Optionally, the adaptation layer sends the aggregated communication results to the computing node or computing device where the target training process resides. For example, the adaptation layer determines the target identification information of the target training process in the historical communication process group based on the first mapping data; and sends the aggregated communication results to the target training process based on the target identification information. Here, the historical communication process group specifically refers to the proxy communication process group, meaning the target identification information is used to indicate the corresponding identification information of the target training process in the proxy communication process group.

[0149] For example, refer to Figure 15 For rank3 in the working communication process group 1502, the identifier information corresponding to rank3 in the proxy communication process group 1501 can be determined as 2 according to the first mapping data. That is, the adapter layer can send the collection communication result to rank2 in the proxy communication process group 1501.

[0150] This embodiment establishes first mapping data, which can meet the backtracking requirements of the identification information of the training process in the working communication process group. This allows the adaptation layer to correctly return the set communication results to the training process based on the first mapping data, which helps to ensure the accuracy of the distribution of set communication results.

[0151] In summary, the technical solution provided in this application deploys an adaptation layer between the training process and the actual communication interface. The adaptation layer reconstructs the working communication process group based on the survival status of the training process participating in the collective communication, and calls the actual communication interface of the working communication process group to obtain the collective communication result, thereby realizing collective communication. This makes the training process unaware of changes to the working communication process group and the actual communication interface, and thus makes the training process unaware of the fault-tolerant process of collective communication.

[0152] Furthermore, by reconstructing the working communication process group based on the survival status of the training processes participating in the ensemble communication, surviving training processes can continue to participate in the ensemble communication, thus avoiding interruptions and ensuring that the distributed training task is not interrupted. This is beneficial for improving the reliability and fault tolerance of the distributed training system corresponding to the distributed training task. Moreover, by eliminating the need to wait for fault recovery before resuming ensemble communication, the latency introduced during fault recovery can be avoided, thereby helping to maintain the training efficiency of the distributed training system.

[0153] In this embodiment of the application, the central controller performs real-time detection of the survival status of multiple training processes participating in the collective communication, so that the adaptation layer can promptly identify the faulty process. This embodiment of the application may also include the following.

[0154] S41, under the target condition, a status acquisition request is sent to the central controller, which is used to monitor the survival status of the training process.

[0155] The status acquisition request is used to request the liveness status of multiple training processes. For example, the status acquisition request includes the identification information of multiple training processes or the identification information of the proxy communication process group to which multiple training processes belong, so that the central controller can accurately provide the liveness status of multiple training processes. That is, in the target scenario, the adaptation layer obtains the liveness status of multiple training processes participating in the aggregate communication through the status acquisition request.

[0156] The target scenarios include at least one of the following: before calling the actual communication interface of the historical communication process group (i.e., before performing a set communication operation on the training process in the historical communication process group), a communication anomaly is detected during the set communication of the historical communication process group, or the set communication of the historical communication process group fails.

[0157] Communication anomalies can refer to exceptions that cause collective communication to fail, such as the occurrence of a faulty process or network failure. For example, if a faulty process is detected during the collective communication process of a historical communication process group, the adaptation layer will stop the collective communication process of the historical communication process group and reconstruct the working communication process group.

[0158] Collective communication failure can refer to situations such as failing to obtain the collective communication result or obtaining an inaccurate collective communication result. For example, if the adaptation layer determines that collective communication for a historical communication process group has failed, it reconstructs the working communication process group to continue executing the collective communication operation; if the adaptation layer determines that collective communication for a historical communication process group has succeeded, it prepares for the next iteration process without reconstructing the working communication process group.

[0159] In one example, the central controller monitors the liveness status of training processes in a distributed training system in real time. This process may include the following: the central controller determines the liveness status of multiple training processes in real time, and the liveness status indicates whether a training process is inactive. A faulty process is inactive.

[0160] If the training process is inactive, the adaptation layer can determine that the training process has failed and cannot continue to participate in the distributed training task (including aggregate communication). If the training process is alive, the adaptation layer can determine that the training process has not failed and can continue to participate in the distributed training task. This application does not limit the method for determining the survival status.

[0161] In one example, the central controller can use a heartbeat mechanism to detect the survival status of multiple training processes in real time. For example, the central controller maintains heartbeats with each of the multiple training processes, that is, each training process sends a heartbeat signal to the central controller at a set interval to inform the central controller that it is alive. The set interval can be set and adjusted according to actual usage requirements, and this embodiment does not limit it.

[0162] For example, for the first training process among multiple training processes, the central controller determines that the state of the first training process is inactive if it does not receive a heartbeat signal from the first training process for a set period of time. The first computing node can be any one of the multiple computing nodes. The set period can be set and adjusted according to actual usage requirements, and this embodiment does not limit it.

[0163] For example, refer to Figure 12 The distributed training system also includes a central controller 1201. If the central controller 1201 does not receive a heartbeat signal from the training process 2 for a set period of time, it can determine that the survival status of the training process 2 is inactive.

[0164] In one example, the central controller determines that the first training process is inactive upon receiving an abnormal signal from it. The abnormal signal is used to inform the central controller that a malfunction has occurred.

[0165] For example, refer to Figure 12 If training process 2 encounters a fault, it sends an abnormal signal to the central controller 1201 to inform itself of the fault. Optionally, if there are inactive training processes among multiple training processes, the central controller can determine that there is a faulty process among the multiple training processes and send the member status data to the adaptation layer.

[0166] Optionally, the central controller can maintain a member list to record the training processes participating in the distributed training task. If a faulty process exists among multiple training processes, the central controller will remove the faulty process from the member list.

[0167] The central controller can generate member status data based on the member list, or it can generate member status data based on the survival status of multiple training processes. This embodiment of the application does not limit this. The member status data is used to indicate the survival status of multiple training processes participating in the ensemble communication. Upon receiving a status acquisition request, the central controller sends the member status data to the adaptation layer.

[0168] S42 receives member status data sent by the central controller.

[0169] After receiving member status data, the adaptation layer determines the survival status of multiple training processes based on the member status data, and then determines whether there are faulty processes (i.e. inactive training processes) in the historical communication process group, so as to determine whether it is necessary to reconstruct the working communication process group.

[0170] This embodiment uses a central controller to monitor the survival status of the training process in real time, enabling the adaptation layer to achieve fault tolerance for ensemble communication (i.e., reconstruct the working communication process group to ensure uninterrupted ensemble communication), thereby achieving uninterrupted ensemble communication.

[0171] In this embodiment, after the faulty process recovers, the faulty process can be reconstructed as a new process to allow the recovered faulty process to continue participating in the distributed training tasks and aggregate communication in subsequent stages. Alternatively, the faulty process can be directly isolated, or other training processes can be used as new processes to replace the faulty process in the reconstruction of the working communication process group. This is beneficial for quickly recovering the aggregate communication in this stage, and this embodiment does not limit the scope of this application.

[0172] In one example, after a faulty process is isolated, the distributed training system can call a new process to replace the faulty process and continue executing the distributed training task, thereby ensuring uninterrupted communication and parallelism of the distributed training system, and thus maintaining the performance and efficiency of distributed training. Embodiments of this application may also include the following:

[0173] S51, if a new process is detected to replace the faulty process, a recovery worker process group is created based on the new process, and the recovery worker process group includes the new process.

[0174] The recovery working process group serves as the working communication process group for aggregate communication in subsequent iterations. Optionally, the adaptation layer can insert a new process into the working communication process group for a failed process to obtain a recovery working process group. For example, this process may include: upon detecting a new process to replace the failed process, adjusting the communication topology of the working communication process group based on the new process to obtain an adjusted communication topology. The adjusted communication topology indicates the communication relationships of each training process in the recovery working process group in aggregate communication. The recovery working process group is then constructed based on the adjusted communication topology.

[0175] Optionally, if there is a backup training process, the backup training process can be used as a new process to replace the faulty process; if there is not enough backup training processes, one can wait for a backup training process to appear and then use the backup training process as a new process to replace the faulty process. This application does not limit this.

[0176] In this embodiment, if a failed process occurs during a stage when backup training processes are available, the new process corresponding to that failed process will not participate in the aggregated communication of that stage. Instead, it will participate in the distributed training task and aggregated communication in the next stage, thereby ensuring uninterrupted aggregated communication and thus uninterrupted distributed training. If there are insufficient backup training processes, newly added processes can participate in the aggregated communication of the target stage, which is the first stage the new process joins.

[0177] After a new process participates in the distributed training task, it proactively informs the central controller of its identity information and establishes a heartbeat with the central controller. Once the heartbeat is established, the new process sends heartbeat signals to the central controller at set intervals.

[0178] Optionally, the adaptation layer can update the communication topology based solely on the communication relationships related to the faulty process within the communication topology of the working communication process group, according to the new process, to obtain the adjusted communication topology. For example, the process of obtaining the adjusted communication topology may include the following:

[0179] (1) Based on the communication topology, identify at least one associated process that has a communication relationship with the faulty process.

[0180] Optionally, the adaptation layer can directly record at least one associated process of the faulty process. The method for determining the associated process is the same as described in the above embodiments, and will not be repeated here.

[0181] (2) For the communication topology of the working communication process group, according to the communication relationship between the faulty process and at least one associated process, establish the communication relationship between the new process and at least one associated process to obtain the adjusted communication topology.

[0182] Optionally, for the communication topology of the working communication process group, the adaptation layer can directly replace the faulty process in the communication topology with a new process to obtain the adjusted communication topology.

[0183] For example, when the communication topology of the working communication process group is a ring structure, and when the i-th training process is a faulty process, for the communication topology of the working communication process group, the adaptation layer clears the communication relationship between the (i-1)-th training process and the (i+1)-th training process, and establishes communication relationships between the new process and the (i-1)-th and (i+1)-th training processes respectively, to obtain the adjusted communication topology.

[0184] In the case that the first training process is a faulty process, for the communication topology of the working communication process group, the adaptation layer clears the communication relationship between the m-th training process and the second training process, establishes a communication relationship between the m-th training process and the new process, and establishes a communication relationship between the new process and the second training process, thus obtaining the adjusted communication topology.

[0185] For example, refer to Figure 12 Upon detecting a new process 2, the adaptation layer clears the communication relationship between rank1 and rank3, establishes a communication relationship between the new process 2 and rank3, and establishes a communication relationship between rank1 and the new process 2, resulting in an adjusted communication topology. The communication relationship between rank0 and rank1 remains unchanged, as does the communication relationship between rank3 and rank0.

[0186] This embodiment adjusts the communication relationships locally based on the communication topology of the working communication process group, without making global adjustments to the communication relationships. This allows for rapid adjustment of the communication topology, which improves the efficiency of the adjustment and reduces the impact on the aggregated communication.

[0187] The adaptation layer determines the adjusted communication topology as the communication topology for the recovery working process group, and constructs the recovery working process group based on the training process indicated by the adjusted communication topology.

[0188] In a feasible example, the adaptation layer creates a recovery worker process group based on the historical communication process group. This could involve replacing faulty processes in the historical communication process group with new processes to obtain the recovery worker process group. Furthermore, the actual communication interface of the historical communication process group is determined as the actual communication interface of the recovery worker process group; alternatively, the actual communication interface of the recovery worker process group is reconstructed.

[0189] This embodiment supports directly replacing faulty processes in historical communication process groups with new processes to build recovery working process groups, which helps improve the efficiency of building recovery working process groups.

[0190] S52, Create the actual communication interface for the recovery worker process group. The set communication function corresponding to the actual communication interface of the recovery worker process group is used to perform set communication operations for the training processes in the recovery worker process group.

[0191] The adaptation layer creates the actual communication interface for the recovery worker process group based on its communication topology. This actual communication interface is implemented as a complete, standard communication process group interface.

[0192] S53, call the actual communication interface corresponding to the recovery worker process group, perform a set communication operation, and obtain the set communication result.

[0193] Upon receiving a collective communication request from each training process in the recovery working process group, the adaptation layer calls the actual communication interface corresponding to the recovery working process group. Through the collective communication function (such as a communication operator) corresponding to the actual communication interface, it performs collective communication operations on each training process in the recovery working process group, thereby achieving data synchronization between the training processes in the recovery working process group.

[0194] S54 sends the ensemble communication results to the training process in the recovery worker process group.

[0195] The process of obtaining and distributing the collective communication results corresponding to the recovery working process group is the same as that of obtaining and distributing the collective communication results corresponding to the working communication process group in the above embodiment, and will not be described again here.

[0196] In this embodiment, during distributed training, the working communication process group is dynamically updated based on the survival status of each training process to maintain uninterrupted ensemble communication and the training scale of the distributed training system until the distributed training task is completed. This helps avoid training state rollback due to training process failures and the resulting loss of computing power, thereby ensuring the performance of the distributed training system.

[0197] In some embodiments, reference Figure 16 Taking the data parallel strategy as an example, the technical solution provided in this application embodiment may also include the following contents.

[0198] S61, the process before the start of group communication.

[0199] After determining the t training processes participating in the distributed training task, according to the data parallelism strategy, the t training processes are divided into s training process groups, and the s training process groups execute the distributed training task in parallel. Furthermore, for each of the s training process groups, multiple communication process groups are constructed, each containing s training processes. The training processes within the same communication process group participate in the same set of communication, where s is an integer greater than 1.

[0200] The adaptation layer identifies multiple communication process groups as proxy communication process groups and declares them in the distributed training system.

[0201] The s training process groups synchronize training state data (such as machine learning model parameters) and prepare training samples to complete the initialization of the distributed training system. After the distributed training system completes initialization, it begins executing distributed training tasks.

[0202] For example, after distributed training begins, each training process performs normal local training on the machine learning model (forward computation and backward computation, etc.).

[0203] S62, the adaptation layer obtains the survival status of the training process.

[0204] This embodiment takes any one of the multiple proxy communication process groups as an example. Before performing the collective communication operation on the proxy communication process group, the adaptation layer queries the central controller to check the survival status of each training process in the proxy communication process group.

[0205] Based on the survival status of each training process, determine whether the working process group needs to be rebuilt. For example, if it is determined that there is a faulty process in the proxy communication process group, execute step S63; if it is determined that there is no faulty process in the proxy communication process group, execute step S64.

[0206] S63, Rebuild the working communication process group.

[0207] The adaptation layer reconstructs the working communication process group to replace the proxy communication process group based on the surviving training processes in the proxy communication process group, and creates the actual communication interface of the working communication process group.

[0208] After the working communication process group is rebuilt, step S64 is executed.

[0209] S64 executes a set communication operation.

[0210] If it is determined that there are no faulty processes in the proxy communication process group, the proxy communication process group is designated as the working communication process group, and the collection communication operation is performed by calling the actual communication interface of the proxy communication process group.

[0211] Optionally, if it is determined that there is a faulty process in the agent communication process group, a collection communication operation is performed by calling the actual communication interface of the working communication process group.

[0212] The adaptation layer determines whether the collection communication is successful. If the collection communication is successful, step S65 is executed; if the collection communication fails, step S62 is executed to re-execute the collection communication process for the current working communication process group.

[0213] S65, the process after the collection communication is completed.

[0214] The adaptation layer distributes the ensemble communication results to the training processes in the worker communication process group to complete parameter and gradient synchronization. Based on the ensemble communication results, the training processes update their local training state data, such as updating the optimizer state and local model parameters, in preparation for executing the next round of distributed training tasks.

[0215] The optimizer is used to train the machine learning model, and the optimizer state is the training state data. Local model parameters refer to the model parameters of the machine learning model stored on the compute node.

[0216] After all training processes have completed the optimizer state and model parameter updates, the master compute node checks whether distributed training is complete. If distributed training is complete, the master compute node controls all training processes to stop the distributed training task; if distributed training is incomplete, proceed to step S62.

[0217] If a faulty process exists, and the faulty process recovers during subsequent distributed training, the working communication process group can be reconstructed for the faulty process to perform collective communication operations.

[0218] In summary, by adopting the technical solution provided in the embodiments of this application, an adaptation layer is deployed between the training process and the actual communication interface. The adaptation layer reconstructs the working communication process group based on the survival status of the training process participating in the collective communication, and calls the actual communication interface of the working communication process group to obtain the collective communication result, thereby realizing the collective communication. This makes the training process unaware of changes to the working communication process group and the actual communication interface, and thus makes the training process unaware of the fault-tolerant process of the collective communication.

[0219] Furthermore, by reconstructing the working communication process group based on the survival status of the training processes participating in the ensemble communication, surviving training processes can continue to participate in the ensemble communication, thus avoiding interruptions and ensuring that the distributed training task is not interrupted. This is beneficial for improving the reliability and fault tolerance of the distributed training system corresponding to the distributed training task. Moreover, by eliminating the need to wait for fault recovery before resuming ensemble communication, the latency introduced during fault recovery can be avoided, thereby helping to maintain the training efficiency of the distributed training system.

[0220] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0221] refer to Figure 17 This diagram illustrates a block diagram of a collective communication device provided in one possible implementation of this application. The device has the functionality to implement the method examples described above; this functionality can be implemented in hardware or by hardware executing corresponding software. The device can be the computer device described above, or it can be located within a computer device. The device is situated between the training process and the actual communication interface; the training process is used to execute distributed training tasks, and the actual communication interface is used to execute collective communication operations. Figure 17As shown, the device 1700 includes: a survival status acquisition module 1701, a communication group reconstruction module 1702, a communication result acquisition module 1703, and a communication result sending module 1704.

[0222] The survival status acquisition module 1701 is used to acquire the survival status of multiple training processes participating in the set communication.

[0223] The communication group reconstruction module 1702 is used to reconstruct a working communication process group based on the currently surviving training process when it is determined that a faulty process exists according to the survival status. The working communication process group is used to replace the historical communication process group used before the fault.

[0224] The communication result acquisition module 1703 is used to call the actual communication interface corresponding to the working communication process group, execute the collective communication operation, and obtain the collective communication result.

[0225] The communication result sending module 1704 is used to send the set communication results to the training process in the working communication process group.

[0226] In some embodiments, the communication result sending module 1704 is configured to: Based on the working communication process group and the historical communication process group, first mapping data is determined. The first mapping data is used to indicate the identification information of each training process in the working communication process group in the historical communication process group and the working communication process group, respectively. For the target training process in the working communication process group, the set communication result is sent to the target training process according to the first mapping data.

[0227] In some embodiments, the communication result sending module 1704 is further configured to: Based on the first mapping data, the target identification information of the target training process in the historical communication process group is determined; Based on the target identification information, the set communication result is sent to the target training process.

[0228] In some embodiments, the communication result acquisition module 1703 is configured to: The actual communication interface is invoked to execute the set communication operation and obtain the initial set communication result. The initial set communication result is the result obtained by executing the set communication operation on the training process in the working communication process group. The communication operator corresponding to the actual communication interface is determined, and the communication operator is used to process the data that each training process in the working communication process group needs to synchronize; Based on the operation type corresponding to the communication operator, an optimization operation is performed on the initial set communication result to obtain the set communication result.

[0229] In some embodiments, the communication result acquisition module 1703 is further configured to: When the operation type corresponding to the communication operator is the first type, an averaging operation is performed on the initial set communication result to obtain the basic set communication result; the set communication result is obtained based on the basic set communication result and the number of processes in the historical communication process group, wherein the first type includes: summation operation; Alternatively, if the operation type corresponding to the communication operator is the second type, the initial set communication result is determined as the set communication result, and the second type includes at least one of the following: calculating the arithmetic mean, calculating the maximum value, calculating the minimum value, bitwise AND, bitwise OR, and bitwise XAND; Alternatively, if the operation type corresponding to the communication operator is the third type, a scaling factor is determined based on the number of processes in the working communication process group; the initial set communication result is adjusted based on the scaling factor to obtain the set communication result, wherein the third type includes a product operation.

[0230] In some embodiments, the apparatus 1700 further includes: an acquisition request generation module and a status data receiving module (not shown in the figure).

[0231] The request generation module is used to send a status acquisition request to the central controller under the target condition. The central controller is used to monitor the survival status of the training process.

[0232] A status data receiving module is used to receive member status data sent by the central controller. The member status data is used to indicate the survival status of multiple training processes participating in the collective communication. The target situation includes at least one of the following: a communication anomaly is detected before the actual communication interface of the historical communication process group is called, a communication anomaly is detected during the collective communication of the historical communication process group, or the collective communication of the historical communication process group fails.

[0233] In some embodiments, the device 1700 further includes a communication interface creation module (not shown in the figure).

[0234] The communication group reconstruction module 1702 is further configured to, upon detecting a new process for replacing the faulty process, create a recovery working process group based on the new process, the recovery working process group including the new process.

[0235] The communication interface creation module 1707 is used to create the actual communication interface of the recovery working process group. The set communication function corresponding to the actual communication interface of the recovery working process group is used to perform set communication operations for the training process in the recovery working process group.

[0236] The communication result acquisition module 1703 is also used to call the actual communication interface corresponding to the recovery working process group, execute the collection communication operation, and obtain the collection communication result; The communication result sending module 1704 is also used to send the set communication result to the training process in the recovery working process group.

[0237] In some embodiments, the communication group reconstruction module 1702 is further configured to replace the faulty process in the historical communication process group with the new process to obtain the recovery working process group.

[0238] In some embodiments, the faulty process is in an inactive state; wherein, for the first training process among the plurality of training processes, the central controller is configured to determine that the state of the first training process is the inactive state if it does not receive a heartbeat signal from the first training process for a set period of time; or, the central controller is configured to determine that the state of the first training process is the inactive state if it receives an abnormal signal from the first training process.

[0239] In some embodiments, the communication group reconstruction module 1702 is further configured to: Based on the communication topology corresponding to the historical communication process group, at least one associated process that has a communication relationship with the faulty process is determined, wherein the communication topology is used to indicate the communication relationship of each process in the historical communication process group in the collective communication. For the communication topology, the communication relationship between the faulty process and the at least one associated process is cleared, and the communication relationship between the remaining processes is adjusted to obtain the updated communication topology; Based on the updated communication topology, create the working communication process group; Based on the work communication process group, create the actual communication interface for the work communication process group.

[0240] In some embodiments, the communication topology is a ring topology, the number of processes in the historical communication process group is m, the m-th training process in the historical communication process group has a communication relationship with the first training process, and the i-th training process in the historical communication process group has a communication relationship with the (i-1)-th training process and the (i+1)-th training process, respectively, where i is an integer greater than 1 and less than m. The communication group reconstruction module 1702 is further configured to, in the case that the i-th training process is the faulty process, clear the communication relationship between the i-th training process and the (i-1)-th and (i+1)-th training processes respectively, and establish a communication relationship between the (i-1)-th and (i+1)-th training processes to obtain the updated communication topology.

[0241] In summary, the technical solution provided in this application deploys an adaptation layer between the training process and the actual communication interface. The adaptation layer reconstructs the working communication process group based on the survival status of the training process participating in the collective communication, and calls the actual communication interface of the working communication process group to obtain the collective communication result, thereby realizing collective communication. This makes the training process unaware of changes to the working communication process group and the actual communication interface, and thus makes the training process unaware of the fault-tolerant process of collective communication.

[0242] Furthermore, by reconstructing the working communication process group based on the survival status of the training processes participating in the ensemble communication, surviving training processes can continue to participate in the ensemble communication, thus avoiding interruptions and ensuring that the distributed training task is not interrupted. This is beneficial for improving the reliability and fault tolerance of the distributed training system corresponding to the distributed training task. Moreover, by eliminating the need to wait for fault recovery before resuming ensemble communication, the latency introduced during fault recovery can be avoided, thereby helping to maintain the training efficiency of the distributed training system.

[0243] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0244] Please refer to Figure 18 This is a simplified structural block diagram of a computer device 1800 provided in one possible implementation of this application. The computer device 1800 can be implemented as the aforementioned computing node or master computing node. The computer device 1800 can be used to implement the aggregated communication method provided in the above embodiments.

[0245] Typically, computer device 1800 includes a processor 1801 and a memory 1802.

[0246] Processor 1801 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1801 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1801 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1801 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1801 may also include an AI processor for handling computational operations related to machine learning.

[0247] The memory 1802 may include one or more computer-readable storage media, which may be non-transitory. The memory 1802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1802 are used to store a computer program configured to be executed by one or more processors to implement the aforementioned collective communication method.

[0248] Those skilled in the art will understand that Figure 18 The structure shown does not constitute a limitation on the computer device 1800, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0249] Optionally, the computer device 1800 may be a server, a server cluster, an artificial intelligence computing cluster, a cloud computing cluster, etc., wherein the artificial intelligence computing cluster may also be referred to as an intelligent computing cluster or a smart computing cluster, and this application does not limit it.

[0250] In some embodiments, a chip product is also provided, wherein a computer program is stored therein, the computer program being loaded and executed by a processor to implement the above-described collection communication method.

[0251] In some embodiments, a computer-readable storage medium is also provided, wherein a computer program is stored therein, which implements the above-described collection communication method when executed by a processor of a computer device.

[0252] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0253] In some embodiments, a computer program product is also provided, the computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the above-described collection communication method.

[0254] It should be understood that "multiple" as used herein refers to two or more. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order between steps. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this approach.

[0255] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A collective communication method, characterized in that, The method is executed by an adaptation layer, which is located between the training process and the actual communication interface. The training process is used to execute distributed training tasks, and the actual communication interface is used to execute ensemble communication operations. The method includes: Obtain the liveness status of multiple training processes participating in the ensemble communication; If a faulty process is determined based on the survival status, a working communication process group is reconstructed based on the currently surviving training process, and the working communication process group is used to replace the historical communication process group used before the fault. The actual communication interface corresponding to the working communication process group is invoked to execute the collective communication operation and obtain the collective communication result; The results of the aggregated communication are sent to the training process in the working communication process group.

2. The method according to claim 1, characterized in that, Sending the aggregate communication results to the training process in the working communication process group includes: Based on the working communication process group and the historical communication process group, first mapping data is determined. The first mapping data is used to indicate the identification information of each training process in the working communication process group in the historical communication process group and the working communication process group, respectively. For the target training process in the working communication process group, the set communication result is sent to the target training process according to the first mapping data.

3. The method according to claim 2, characterized in that, The step of sending the set communication result to the target training process based on the first mapping data includes: Based on the first mapping data, the target identification information of the target training process in the historical communication process group is determined; Based on the target identification information, the set communication result is sent to the target training process.

4. The method according to claim 1, characterized in that, The step of calling the actual communication interface corresponding to the working communication process group, executing the collective communication operation, and obtaining the collective communication result includes: The actual communication interface is invoked to perform the set communication operation and obtain the initial set communication result. The initial set communication result is the result obtained by performing the set communication operation on the training process in the working communication process group. The communication operator corresponding to the actual communication interface is determined, and the communication operator is used to process the data that each training process in the working communication process group needs to synchronize; Based on the operation type corresponding to the communication operator, an optimization operation is performed on the initial set communication result to obtain the set communication result.

5. The method according to claim 4, characterized in that, The step of performing an optimization operation on the initial set communication result according to the operation type corresponding to the communication operator to obtain the set communication result includes: When the operation type corresponding to the communication operator is the first type, an averaging operation is performed on the initial set communication result to obtain the basic set communication result; the set communication result is obtained based on the basic set communication result and the number of processes in the historical communication process group, wherein the first type includes: summation operation; or, When the operation type corresponding to the communication operator is the second type, the initial set communication result is determined as the set communication result. The second type includes at least one of the following: calculating the arithmetic mean, calculating the maximum value, calculating the minimum value, bitwise AND, bitwise OR, and bitwise XAND. or, When the operation type corresponding to the communication operator is the third type, the scaling factor is determined according to the number of processes in the working communication process group; the initial set communication result is adjusted according to the scaling factor to obtain the set communication result, wherein the third type includes the product operation.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Under the target condition, a status acquisition request is sent to the central controller, which is used to monitor the survival status of the training process; Receive member status data sent by the central controller, the member status data being used to indicate the survival status of multiple training processes participating in the ensemble communication; The target scenario includes at least one of the following: a communication anomaly is detected before the actual communication interface of the historical communication process group is invoked, a communication anomaly is detected during the collective communication of the historical communication process group, or the collective communication of the historical communication process group fails.

7. The method according to claim 6, characterized in that, The method further includes: If a new process is detected to replace the faulty process, a recovery process group is created based on the new process, the recovery process group including the new process; Create the actual communication interface of the recovery worker process group. The set communication function corresponding to the actual communication interface of the recovery worker process group is used to perform set communication operations for the training process in the recovery worker process group. The actual communication interface corresponding to the recovery process group is invoked to perform the collection communication operation and obtain the collection communication result; The collected communication results are sent to the training process in the recovery working process group.

8. The method according to claim 7, characterized in that, The step of creating a recovery worker process group based on the new process includes: The faulty process in the historical communication process group is replaced with the new process to obtain the recovery process group.

9. The method according to claim 6, characterized in that, The faulty process is inactive; Specifically, for the first training process among the plurality of training processes, the central controller is configured to determine the state of the first training process as the inactive state if it does not receive a heartbeat signal from the first training process for a set period of time; or, the central controller is configured to determine the state of the first training process as the inactive state if it receives an abnormal signal from the first training process.

10. The method according to any one of claims 1 to 5, characterized in that, The reconstruction of the working communication process group based on the currently surviving training processes includes: Based on the communication topology corresponding to the historical communication process group, at least one associated process that has a communication relationship with the faulty process is determined, wherein the communication topology is used to indicate the communication relationship of each process in the historical communication process group in the collective communication. For the communication topology, the communication relationship between the faulty process and the at least one associated process is cleared, and the communication relationship between the remaining processes is adjusted to obtain the updated communication topology; Based on the updated communication topology, create the working communication process group; Based on the work communication process group, create the actual communication interface for the work communication process group.

11. The method according to claim 10, characterized in that, The communication topology is a ring topology. The number of processes in the historical communication process group is m. The m-th training process in the historical communication process group has a communication relationship with the first training process. The i-th training process in the historical communication process group has a communication relationship with the (i-1)-th training process and the (i+1)-th training process, respectively. i is an integer greater than 1 and less than m. The step of clearing the communication relationship between the faulty process and the at least one associated process, and adjusting the communication relationship between the remaining processes to obtain the updated communication topology, includes: In the case that the i-th training process is the faulty process, for the communication topology, the communication relationship between the i-th training process and the (i-1)-th and (i+1)-th training processes is cleared, and a communication relationship is established between the (i-1)-th and (i+1)-th training processes to obtain the updated communication topology.

12. A collective communication device, characterized in that, The device is located between the training process and the actual communication interface, wherein the training process is used to execute distributed training tasks, and the actual communication interface is used to execute aggregate communication operations; the device includes: The survival status acquisition module is used to acquire the survival status of multiple training processes participating in the ensemble communication; The communication group reconstruction module is used to reconstruct a working communication process group based on the currently surviving training process when it is determined that there is a faulty process according to the survival status. The working communication process group is used to replace the historical communication process group used before the fault. The communication result acquisition module is used to call the actual communication interface corresponding to the working communication process group, execute the collective communication operation, and obtain the collective communication result. The communication result sending module is used to send the set communication results to the training process in the working communication process group.

13. A chip product, characterized in that, The chip product stores a computer program, which is loaded and executed by a processor to implement the method as described in any one of claims 1 to 11.

14. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the method as claimed in any one of claims 1 to 11.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the method as described in any one of claims 1 to 11.

16. A computer program product, characterized in that, The computer program product includes a computer program that is executed by a processor to implement the method as described in any one of claims 1 to 11.

Images