Data parallel communication method and device in distributed training, storage medium and program product

Abstract

The invention relates to a data parallel communication method and device in distributed training, a storage medium and a program product. The method comprises the following steps: for any computing device in a computing device cluster participating in distributed training of a target model, obtaining global latest parameters of the target model through global collection operation in a forward propagation process, according to the training data subsets corresponding to the computing devices and the global latest parameters of the target model, forward propagation calculation is carried out, loss values corresponding to the computing devices are obtained, and different computing devices in the computing device cluster carry out forward propagation calculation based on different training data subsets; and for any computing device, calculating a gradient corresponding to the computing device based on the loss value corresponding to the computing device in a back propagation process, and transmitting the gradient corresponding to the computing device to other computing devices in the computing device cluster through reduction and dispersion operation. The communication overhead can be reduced, and the efficiency of large-scale training can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to a data parallel communication method in distributed training, a data parallel communication device in distributed training, a non-volatile computer-readable storage medium, and a computer program product. Background Art

[0002] With the continuous increase in the scale and complexity of deep learning models, the training data and the number of parameters have grown exponentially. The data parallel method accelerates model training by partitioning the training data across multiple computing devices. The core of its communication lies in the synchronous update of gradients, which usually uses the All-Reduce operator to complete the aggregation and distribution of gradients on each computing device. The specific process includes:

[0003] Forward computation: Each computing device uses the allocated training data for forward computation to obtain the output result of the model (e.g., Y = XW);

[0004] Loss calculation: Calculate the loss value based on the model output and the actual label;

[0005] Backward computation: Based on the loss value, perform backpropagation on the model parameters to calculate the gradients (e.g., dW = XT * dY);

[0006] Gradient aggregation: Aggregate the gradients on each computing device through the All-Reduce operator. Among them, All-Reduce is a commonly used parallel communication operation that adds up the gradients of all computing devices and distributes the result to all computing devices to share and synchronize the gradients among all computing devices;

[0007] Parameter update: Each computing device updates the model parameters according to the aggregated gradients.

[0008] Although this method has been widely used in parallel training, it has the following main problems:

[0009] Low communication efficiency: As the scale of training data and the number of computing devices increase, the global reduction communication time in the gradient aggregation process becomes longer and longer. In large-scale training scenarios, the communication time occupies a significant proportion, seriously affecting the training efficiency.

[0010] Low utilization rate of computing resources: The All-Reduce operation usually starts after the backward computation is completed. That is, the synchronization operation of the All-Reduce operator requires all computing devices to wait, which means that the computing devices are idle while waiting for the communication to complete and do not fully utilize the computing resources.

[0011] Conflicts in multi-parallel strategies: When combining complex parallel strategies such as Tensor Parallel and Pipeline Parallel, the communication requirements of data parallel communication conflict with those of other parallel strategies, resulting in idle computing resources and further reducing training efficiency. For example, the communication operations in tensor parallel backpropagation and data parallel communication may occur simultaneously, causing conflicts and delays. Summary of the Invention

[0012] In view of this, the present disclosure provides a technical solution for data parallel communication in distributed training.

[0013] According to one aspect of the present disclosure, there is provided a method for data parallel communication in distributed training, including:

[0014] For any computing device in a cluster of computing devices participating in the distributed training of a target model, during the forward propagation process, obtain the globally latest parameters of the target model through a global collection operation, and perform forward propagation calculation based on the training data subset corresponding to the computing device and the globally latest parameters of the target model to obtain the loss value corresponding to the computing device, where different computing devices in the cluster of computing devices perform forward propagation calculations based on different training data subsets;

[0015] For any of the computing devices, during the backpropagation process, calculate the gradient corresponding to the computing device based on the loss value corresponding to the computing device, and transmit the gradient corresponding to the computing device to other computing devices in the cluster of computing devices through a reduce-scatter operation.

[0016] In a possible implementation, the transmitting the gradient corresponding to the computing device to other computing devices in the cluster of computing devices through a reduce-scatter operation includes:

[0017] Transmit the gradient corresponding to the computing device to other partial computing devices in the cluster of computing devices through a reduce-scatter operation.

[0018] In a possible implementation, for any computing device in a cluster of computing devices participating in the distributed training of a target model, obtaining the globally latest parameters of the target model through a global collection operation during the forward propagation process includes:

[0019] For any computing device in a cluster of computing devices participating in the distributed training of a target model, during the forward propagation process, respectively obtain the locally latest parameters of the target model from other individual computing devices in the cluster of computing devices through a global collection operation;

[0020] Determine the global up-to-date parameters of the target model based on the local up-to-date parameters of the target model corresponding to the computing device and the local up-to-date parameters of the target model corresponding to each of the other computing devices.

[0021] In a possible implementation, the transmitting the gradients corresponding to the computing device to other computing devices in the computing device cluster through a reduction-scatter operation includes:

[0022] For a first module of the target model, in response to the first module not being the first module of the target model, during the backpropagation calculation of a second module, transmit the gradients of the first module corresponding to the computing device to other computing devices in the computing device cluster through a reduction-scatter operation, where the first module performs backpropagation calculation earlier than the second module;

[0023] Alternatively, for the first module, in response to the first module being the first module of the target model, during the forward propagation of the target model, transmit the gradients of the first module corresponding to the computing device to other computing devices in the computing device cluster through a reduction-scatter operation.

[0024] In a possible implementation, the target model adopts a Transformer network structure, and the modules in the target model include a query-key-value linear transformation module, a fast attention module, a dimensionality reduction projection module, a first fully connected layer, and a second fully connected layer.

[0025] In a possible implementation, for a first module of the target model, in response to the first module not being the first module of the target model, during the backpropagation calculation of a second module, transmit the gradients of the first module corresponding to the computing device to other computing devices in the computing device cluster through a reduction-scatter operation, where the first module performs backpropagation calculation earlier than the second module, includes at least one of the following:

[0026] During the calculation of the input gradients of the dimensionality reduction projection module, transmit the parameter gradients of the second fully connected layer corresponding to the computing device to other computing devices in the computing device cluster through a reduction-scatter operation;

[0027] During the backpropagation calculation of the fast attention module, transmit the parameter gradients of the first fully connected layer and the parameter gradients of the dimensionality reduction projection module corresponding to the computing device to other computing devices in the computing device cluster through a reduction-scatter operation;

[0028] For a query-key-value linear transformation module that is not the first layer, during the calculation of the input gradient of the second fully connected layer in the previous layer, the parameter gradient of the query-key-value linear transformation module corresponding to the computing device is transmitted to other computing devices in the computing device cluster through a reduction-scatter operation.

[0029] In a possible implementation, for the first module, in response to the first module being the first module of the target model, during the forward propagation of the target model, the gradient of the first module corresponding to the computing device is transmitted to other computing devices in the computing device cluster through a reduction-scatter operation, including:

[0030] For the query-key-value linear transformation module of the first layer, during the calculation of the embedding vector, the parameter gradient of the query-key-value linear transformation module corresponding to the computing device is transmitted to other computing devices in the computing device cluster through a reduction-scatter operation.

[0031] In a possible implementation, obtaining the globally latest parameters of the target model through a global gather operation includes:

[0032] During the forward propagation calculation process of the third module of the target model, obtaining the globally latest parameters of the fourth module of the target model through a global gather operation, where the fourth module is the next module of the third module.

[0033] According to another aspect of the present disclosure, there is provided a data parallel communication device in distributed training, including:

[0034] A forward propagation processing module, configured to, for any computing device in the computing device cluster participating in the distributed training of the target model, during the forward propagation process, obtain the globally latest parameters of the target model through a global gather operation, and perform forward propagation calculation based on the training data subset corresponding to the computing device and the globally latest parameters of the target model to obtain the loss value corresponding to the computing device, where different computing devices in the computing device cluster perform forward propagation calculation based on different training data subsets;

[0035] A backward propagation processing module, configured to, for the any computing device, during the backward propagation process, calculate the gradient corresponding to the computing device based on the loss value corresponding to the computing device, and transmit the gradient corresponding to the computing device to other computing devices in the computing device cluster through a reduction-scatter operation.

[0036] In a possible implementation, the backward propagation processing module is configured to:

[0037] Transmit the gradients corresponding to the computing device to other computing devices in the computing device cluster through a reduction scatter operation.

[0038] In a possible implementation, the forward propagation processing module is configured to:

[0039] For any computing device in the computing device cluster participating in the distributed training of the target model, during the forward propagation process, obtain the local latest parameters of the target model from each of the other computing devices in the computing device cluster through a global collection operation;

[0040] Determine the global latest parameters of the target model according to the local latest parameters of the target model corresponding to the computing device and the local latest parameters of the target model corresponding to each of the other computing devices.

[0041] In a possible implementation,

[0042] The backpropagation processing module is configured to: for the first module of the target model, in response to the first module not being the first module of the target model, during the backpropagation calculation of the second module, transmit the gradients of the first module corresponding to the computing device to other computing devices in the computing device cluster through a reduction scatter operation, where the first module performs backpropagation calculation earlier than the second module;

[0043] Alternatively, the forward propagation processing module is configured to: for the first module, in response to the first module being the first module of the target model, during the forward propagation process of the target model, transmit the gradients of the first module corresponding to the computing device to other computing devices in the computing device cluster through a reduction scatter operation.

[0044] In a possible implementation, the target model adopts a Transformer network structure, and the modules in the target model include a query-key-value linear transformation module, a fast attention module, a dimensionality reduction projection module, a first fully connected layer, and a second fully connected layer.

[0045] In a possible implementation, the backpropagation processing module is used for at least one of the following:

[0046] During the calculation of the input gradients of the dimensionality reduction projection module, transmit the parameter gradients of the second fully connected layer corresponding to the computing device to other computing devices in the computing device cluster through a reduction scatter operation;

[0047] During the backpropagation calculation process of the fast attention module, the parameter gradients of the first fully connected layer corresponding to the computing device and the parameter gradients of the dimensionality reduction projection module are transmitted to other computing devices in the computing device cluster through a reduction-scatter operation;

[0048] For the query-key-value linear transformation module of non-the first layer, during the calculation process of the input gradients of the second fully connected layer of the previous layer, the parameter gradients of the query-key-value linear transformation module corresponding to the computing device are transmitted to other computing devices in the computing device cluster through a reduction-scatter operation.

[0049] In a possible implementation manner, the forward propagation processing module is configured to:

[0050] For the query-key-value linear transformation module of the first layer, during the calculation process of the embedding vector, the parameter gradients of the query-key-value linear transformation module corresponding to the computing device are transmitted to other computing devices in the computing device cluster through a reduction-scatter operation.

[0051] In a possible implementation manner, the forward propagation processing module is configured to:

[0052] During the forward propagation calculation process of the third module of the target model, the globally latest parameters of the fourth module of the target model are obtained through a global gather operation, where the fourth module is the next module of the third module.

[0053] According to another aspect of the present disclosure, there is provided a data parallel communication device in distributed training, including a memory, a processor, and a computer program stored on the memory, and the processor executes the computer program to implement the steps of the above method.

[0054] According to another aspect of the present disclosure, there is provided a non-volatile computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0055] According to another aspect of the present disclosure, there is provided a computer program product, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0056] In the embodiments of the present disclosure, for any computing device in a cluster of computing devices participating in the distributed training of a target model, during the forward propagation process, the globally latest parameters of the target model are obtained through a global collection operation, and forward propagation calculation is performed based on the training data subset corresponding to the computing device and the globally latest parameters of the target model to obtain the loss value corresponding to the computing device. Among them, different computing devices in the cluster of computing devices perform forward propagation calculations based on different training data subsets. For any one of the computing devices, during the backward propagation process, based on the loss value corresponding to the computing device, the gradient corresponding to the computing device is calculated, and the gradient corresponding to the computing device is transmitted to other computing devices in the cluster of computing devices through a reduction-scatter operation. Thus, the global reduction operation is split into a reduction-scatter operation and a global collection operation. The reduction-scatter operation is performed during the backward calculation process, and the global collection operation is performed during the forward calculation process, making the communication execution timing more flexible, capable of overlapping the communication time of data parallel communication with the calculation time, realizing the hiding of communication time, reducing communication overhead, effectively shortening the proportion of communication in the total training time, reducing the impact of communication time on the overall training efficiency, and being able to reduce the waiting time of computing devices during the training process, realizing the full utilization of computing resources, thereby being able to significantly improve the efficiency of large-scale training and being applicable to data parallel scenarios where a large number of computing devices participate.

[0057] Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are included in and constitute a part of this specification, illustrate exemplary embodiments, features, and aspects of the present disclosure together with the specification and are used to explain the principles of the present disclosure.

[0059] Figure 1 The flowchart showing the data parallel communication method in distributed training provided by the embodiments of the present disclosure.

[0060] Figure 2 The block diagram showing the data parallel communication device in distributed training provided by the embodiments of the present disclosure.

[0061] Figure 3 The block diagram of a data parallel communication device 1900 in distributed training shown according to an exemplary embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0062] The following will describe in detail various exemplary embodiments, features, and aspects of the present disclosure with reference to the accompanying drawings. The same reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0063] As used herein, the terms "comprising," "including," "having," or variations thereof are open-ended and include one or more stated features, integers, elements, steps, components, or functions, but do not preclude the existence or addition of one or more other features, integers, elements, steps, components, functions, or groups thereof.

[0064] When an element is referred to as being "connected," "coupled," "responsive," or variations thereof to another element, it can be directly connected, coupled, or responsive to the other element, or intervening elements may be present.

[0065] Although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Thus, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments without departing from the teachings of the inventive concept.

[0066] The word "exemplary" as used herein means "serving as an example, instance, or illustration." Any embodiment so described as "exemplary" need not be construed as preferred or better than other embodiments.

[0067] In addition, for a better illustration of the present disclosure, numerous specific details are given in the following detailed description. Those skilled in the art should understand that the present disclosure can be implemented without some of these specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art are not described in detail so as to highlight the gist of the present disclosure.

[0068] To solve the technical problems similar to those described above, embodiments of the present disclosure provide a data parallel communication method in distributed training. For any computing device in a cluster of computing devices participating in the distributed training of a target model, during the forward propagation process, the global latest parameters of the target model are obtained through a global collection operation, and forward propagation calculation is performed based on the training data subset corresponding to the computing device and the global latest parameters of the target model to obtain the loss value corresponding to the computing device. Among them, different computing devices in the computing device cluster perform forward propagation calculations based on different training data subsets. For any one of the computing devices, during the backward propagation process, based on the loss value corresponding to the computing device, the gradient corresponding to the computing device is calculated, and the gradient corresponding to the computing device is transmitted to other computing devices in the computing device cluster through a reduce-scatter operation. Thus, the global reduction operation is split into a reduce-scatter operation and a global collection operation. The reduce-scatter operation is performed during the backward calculation process, and the global collection operation is performed during the forward calculation process, making the communication execution timing more flexible. It can overlap the communication time of data parallel communication with the calculation time, realize the hiding of communication time, reduce communication overhead, effectively shorten the proportion of communication in the total training time, reduce the impact of communication time on the overall training efficiency, and can reduce the waiting time of computing devices during the training process, realize the full utilization of computing resources, thereby significantly improving the efficiency of large-scale training and being applicable to data parallel scenarios involving a large number of computing devices.

[0069] The following describes in detail the data parallel communication method in distributed training provided by embodiments of the present disclosure with reference to the accompanying drawings.

[0070] Figure 1 The flowchart showing the data parallel communication method in distributed training provided by embodiments of the present disclosure is shown. In a possible implementation manner, the execution subject of the data parallel communication method in distributed training may be a data parallel communication device in distributed training. For example, the data parallel communication method in distributed training may be executed by a terminal device, a server, or other electronic devices. Among them, the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, or a wearable device, etc. In some possible implementation manners, the data parallel communication method in distributed training may be implemented by a processor calling computer-readable instructions stored in a memory. As Figure 1 shown, the data parallel communication method in distributed training includes steps S11 to S12.

[0071] In step S11, for any computing device in the computing device cluster participating in the distributed training of the target model, during the forward propagation process, the global latest parameters of the target model are obtained through a global collection operation, and forward propagation calculation is performed based on the training data subset corresponding to the computing device and the global latest parameters of the target model to obtain the loss value corresponding to the computing device. Among them, different computing devices in the computing device cluster perform forward propagation calculation based on different training data subsets.

[0072] In step S12, for any one of the computing devices, during the backward propagation process, based on the loss value corresponding to the computing device, the gradient corresponding to the computing device is calculated, and the gradient corresponding to the computing device is transmitted to other computing devices in the computing device cluster through a reduce-scatter operation.

[0073] In the embodiments of the present disclosure, the target model may refer to a deep learning model being trained. In distributed training, multiple computing devices can be used for parallel training of the target model. The set of these computing devices can be referred to as a computing device cluster. To make full use of computing resources, each computing device can process a subset of the model training data. This requires a mechanism to ensure that the model parameters on all computing devices are consistent so that they can work together.

[0074] In the embodiments of the present disclosure, the all-reduce operation is replaced by the reduce-scatter operation and the all-gather operation. By replacing the all-reduce operation with the reduce-scatter operation and the all-gather operation, the amount of data communicated each time is reduced, and the communication time is reduced. This improvement enhances the flexibility of communication and provides a basis for adjusting the communication timing.

[0075] Among them, the all-gather operation can be used to obtain the local latest parameters of the target model from other computing devices and aggregate these local latest parameters into the global latest parameters. The global latest parameters are the latest versions of the model parameters on all computing devices. Each computing device can perform forward propagation calculation using its own training data subset and the global latest parameters, and finally obtain the loss value corresponding to the computing device.

[0076] In a possible implementation, for any computing device in the computing device cluster participating in the distributed training of the target model, during the forward propagation process, the global up-to-date parameters of the target model are obtained through a global collection operation, including: for any computing device in the computing device cluster participating in the distributed training of the target model, during the forward propagation process, the local up-to-date parameters of the target model are respectively obtained from other computing devices in the computing device cluster through a global collection operation; according to the local up-to-date parameters of the target model corresponding to the computing device, and the local up-to-date parameters of the target model corresponding to the other computing devices, the global up-to-date parameters of the target model are determined.

[0077] In this implementation, during the forward propagation, the up-to-date parameters (i.e., local up-to-date parameters) of the target model are respectively collected from other computing devices through a global collection operation and summarized, so as to obtain the complete parameter matrix of the target model (i.e., global up-to-date parameters). In this implementation, the loss value calculated by each computing device is based on the global up-to-date parameters, which enables the subsequent backpropagation calculation to correctly update the model parameters.

[0078] In this implementation, the model parameters that originally needed to be synchronized as a whole are decomposed into phased synchronization, reducing the data volume of single-parameter communication and the communication time.

[0079] For example, there are 3 computing devices (Device A, Device B, and Device C), and the training data is divided into 3 subsets, which are respectively processed by these 3 computing devices. Computing devices A, B, and C can respectively calculate the local up-to-date parameters corresponding to their own subsets. Through the global collection operation, computing device A can obtain the local up-to-date parameters of computing devices B and C, and finally obtain the global up-to-date parameters. Similarly, computing devices B and C can also obtain the local up-to-date parameters from other devices and finally obtain the global up-to-date parameters. Computing device A can use the global up-to-date parameters and its own training data subset for forward propagation calculation to obtain the loss value. Computing devices B and C can also respectively use the global up-to-date parameters and their own training data subsets for forward propagation calculation to obtain their respective loss values.

[0080] In the embodiments of the present disclosure, the reduce-scatter operation can be used to transmit gradients to other computing devices.

[0081] In a possible implementation, the transmitting the gradient corresponding to the computing device to other computing devices in the computing device cluster through the reduce-scatter operation includes: transmitting the gradient corresponding to the computing device to other partial computing devices in the computing device cluster through the reduce-scatter operation.

[0082] In this implementation, for any computing device, the gradients corresponding to the computing device can be transmitted to some of the other computing devices in the computing device cluster through a reduction scatter operation, rather than to each computing device. In this implementation, the gradient aggregation operation that originally needed to be completed among all computing devices is decomposed into segmented communication. Each computing device only processes the aggregation of part of the gradients, and finally each computing device holds a part of the aggregated gradient data. By adopting this implementation, the amount of data for a single gradient communication can be further reduced, and the communication time can be significantly reduced.

[0083] In another possible implementation, transmitting the gradients corresponding to the computing device to the other computing devices in the computing device cluster through a reduction scatter operation includes: transmitting the gradients corresponding to the computing device to each of the other computing devices in the computing device cluster through a reduction scatter operation.

[0084] In a possible implementation, transmitting the gradients corresponding to the computing device to the other computing devices in the computing device cluster through a reduction scatter operation includes: for a first module of the target model, in response to the first module not being the first module of the target model, during the backpropagation calculation of a second module, transmitting the gradients of the first module corresponding to the computing device to the other computing devices in the computing device cluster through a reduction scatter operation, where the first module performs backpropagation calculation earlier than the second module; or, for the first module, in response to the first module being the first module of the target model, during the forward propagation of the target model, transmitting the gradients of the first module corresponding to the computing device to the other computing devices in the computing device cluster through a reduction scatter operation.

[0085] In this implementation, when the first module is not the first module of the target model, during the backpropagation calculation of the second module, the reduction scatter operation of the gradients of the first module can be performed simultaneously to transmit the gradients of the first module to other computing devices. Where the first module performs backpropagation calculation earlier than the second module may mean that in the overall process of backpropagation, the calculation time point of the first module is earlier than that of the second module.

[0086] For example, if the target model has three modules (module 1, module 2, module 3) and the order of backpropagation is: module 3 → module 2 → module 1. Then, during the backpropagation calculation of module 2, the reduction scatter operation of the gradients of module 3 can be performed simultaneously to transmit the gradients of module 3 to other computing devices. Or, during the backpropagation calculation of module 1, the reduction scatter operation of the gradients of module 3 can be performed simultaneously to transmit the gradients of module 3 to other computing devices.

[0087] In this implementation, when the first module is the first module of the target model, during the forward propagation of the target model, the gradients of the first module can be transmitted to other computing devices in the computing device cluster through a reduce-scatter operation. Thus, the gradient communication of the first module of the target model can be overlapped with the forward propagation calculation of the target model, hiding the gradient communication time of the first module of the target model.

[0088] For example, the target model has three modules (Module 1, Module 2, Module 3), and the order of backpropagation is: Module 3 → Module 2 → Module 1, where Module 1 is the first module of the target model. Then, during the forward propagation of the target model, the gradients of Module 1 can be transmitted to other computing devices in the computing device cluster through a reduce-scatter operation.

[0089] In this implementation, by adjusting the execution timing of the reduce-scatter operation, the training time is not additionally occupied, and the gradient communication time is hidden in the computing time, enabling the computing devices to always maintain a high-load working state and improving the training efficiency of the target model.

[0090] Backpropagation is an important stage in the training of deep learning models, and its main tasks include calculating the parameter gradients dW and input gradients dX of each layer. In related technologies, these calculations and communications are usually executed serially, and the communication time is difficult to hide, affecting the training efficiency.

[0091] Under the tensor parallel strategy, the reverse communication of the column parallel linear layer has the following characteristics: when calculating the input gradient dX, a global collection operation of the input data needs to be performed to ensure data integrity; when calculating the parameter gradient dW, a reduce-scatter operation needs to be performed on the input gradient dX to disperse and aggregate gradient data; the communication time is tight and it is easy to compete with computing tasks.

[0092] Under the tensor parallel strategy, the reverse communication of the row parallel linear layer has the following characteristics: the communication is relatively idle when calculating the input gradient dX, and the computing device resources are not fully utilized; when calculating the parameter gradient dW, a global collection operation needs to be performed on the calculated parameter gradient dX, and the communication burden is relatively heavy.

[0093] In a possible implementation, the target model adopts a Transformer network structure, and the modules in the target model include a query-key-value linear transformation module, a fast attention module, a dimensionality reduction projection module, a first fully connected layer, and a second fully connected layer.

[0094] The typical modules of the Transformer network perform forward propagation in the following order: query-key-value linear transformation module (qkvlinear) → FlashAttention module → downsampling projection module (downproj) → first fully connected layer (fc1) → second fully connected layer (fc2). Among them, the query-key-value linear transformation module and the first fully connected layer are column parallel linear layers (Column Parallel Linear) that need to handle high-concurrency communication tasks; the downsampling projection module and the second fully connected layer are row parallel linear layers (Row Parallel Linear), and the communication time is relatively controllable.

[0095] In a possible implementation, for the first module of the target model, in response to the first module not being the first module of the target model, during the backpropagation calculation of the second module, the gradient of the first module corresponding to the computing device is transmitted to other computing devices in the computing device cluster through a reduction-scatter operation, where the first module performs backpropagation calculation earlier than the second module, including at least one of the following: during the calculation of the input gradient of the downsampling projection module, the parameter gradient of the second fully connected layer corresponding to the computing device is transmitted to other computing devices in the computing device cluster through a reduction-scatter operation; during the backpropagation calculation of the FlashAttention module, the parameter gradients of the first fully connected layer and the downsampling projection module corresponding to the computing device are transmitted to other computing devices in the computing device cluster through a reduction-scatter operation; for the query-key-value linear transformation module that is not the first layer, during the calculation of the input gradient of the second fully connected layer of the previous layer, the parameter gradient of the query-key-value linear transformation module corresponding to the computing device is transmitted to other computing devices in the computing device cluster through a reduction-scatter operation.

[0096] As an example of this implementation, the reduction-scatter operation of the parameter gradient dW of the second fully connected layer (fc2) can be completed during the calculation of the input gradient of the downsampling projection module (downproj). In this example, by adjusting the execution timing of the reduction-scatter operation of the parameter gradient dW of the second fully connected layer (fc2), the communication of the parameter gradient dW of the second fully connected layer (fc2) can be completely hidden in the backpropagation calculation task of the downsampling projection module (downproj).

[0097] As an example of this implementation, the reduction and scattering operation of the parameter gradient dW of the first fully connected layer (fc1) can be completed in the backpropagation calculation task of the FlashAttention module. In this example, by adjusting the execution timing of the reduction and scattering operation of the parameter gradient dW of the first fully connected layer (fc1), the communication of the parameter gradient dW of the first fully connected layer (fc1) can be completely hidden in the backpropagation calculation task of the FlashAttention module.

[0098] As an example of this implementation, the reduction and scattering operation of the parameter gradient dW of the downsampling projection module (downproj) can be completed in the backpropagation calculation task of the FlashAttention module. In this example, by adjusting the execution timing of the reduction and scattering operation of the parameter gradient dW of the downsampling projection module (downproj), the communication of the parameter gradient dW of the downsampling projection module (downproj) can be completely hidden in the backpropagation calculation task of the FlashAttention module.

[0099] As an example of this implementation, the reduction and scattering operation of the parameter gradient dW of the query-key-value linear transformation module (qkvlinear) other than the first layer can be inserted into the calculation process of the input gradient of the second fully connected layer (fc2) of the previous layer Transformer module. In this example, by adjusting the execution timing of the reduction and scattering operation of the parameter gradient dW of the query-key-value linear transformation module (qkvlinear) other than the first layer, the communication of the parameter gradient dW of the query-key-value linear transformation module (qkvlinear) other than the first layer can be completely hidden in the backpropagation calculation task of the second fully connected layer (fc2) of the previous layer.

[0100] In this implementation, in the scenario of combining multiple parallel strategies, by redesigning the execution order and position of the reduction and scattering operator, conflicts can be avoided and the overall training efficiency can be improved.

[0101] In a possible implementation, for the first module, in response to the first module being the first module of the target model, during the forward propagation of the target model, the gradient of the first module corresponding to the computing device is transmitted to other computing devices in the computing device cluster through a reduction and scattering operation, including: for the query-key-value linear transformation module of the first layer, during the calculation of the embedding vector, the parameter gradient of the query-key-value linear transformation module corresponding to the computing device is transmitted to other computing devices in the computing device cluster through a reduction and scattering operation.

[0102] In this implementation, the reduction-scatter operation of the parameter gradient dW of the query-key-value linear transformation module (qkvlinear) in the first layer can be completed during the calculation of the embedding vector.

[0103] During the forward propagation process, each layer of the network module needs to perform the following tasks: complete calculations such as matrix multiplication using model parameters and input data; trigger communication before calculation to synchronize model parameters or scatter intermediate results according to the requirements of the parallel strategy.

[0104] The forward communication of the Column Parallel Linear layer has the following characteristics: before calculating the matrix multiplication, a global gather operation needs to be performed on the input data to ensure data consistency on each computing device; there is a strict dependency relationship between communication and calculation, and serialization is required.

[0105] The forward communication of the Row Parallel Linear layer has the following characteristics: after completing the matrix multiplication, a reduction-scatter operation needs to be performed on the intermediate results to scatter the data to each computing device for subsequent calculations; this communication is closely related to the subsequent calculations and also needs to be executed serially.

[0106] In a possible implementation, the globally latest parameters of the target model are obtained through a global gather operation, including: during the forward propagation calculation process of the third module of the target model, the globally latest parameters of the fourth module of the target model are obtained through a global gather operation, where the fourth module is the next module of the third module.

[0107] In this implementation, to mask the communication time, the global gather operation is inserted into the calculation task of the previous module to complete.

[0108] As an example of this implementation, the global gather operation of the downsampling projection module (downproj) can be completed during the forward propagation calculation process of the query-key-value linear transformation module (qkvlinear).

[0109] As an example of this implementation, the global gather operation of the first fully connected layer (fc1) can be completed during the forward propagation calculation process of the downsampling projection module (downproj).

[0110] As an example of this implementation, the global gather operation of the second fully connected layer (fc2) can be completed during the forward propagation calculation process of the first fully connected layer (fc1).

[0111] As an example of this implementation manner, during the forward propagation calculation process of the second fully-connected layer (fc2), the global collection operation of the next-layer query-key-value linear transformation module (qkvlinear) can be completed.

[0112] In this implementation manner, the global collection operation of each module overlaps with the forward propagation calculation process of the previous module, enabling the computing device to continue executing tasks without waiting for the communication to complete.

[0113] The data parallel communication method in distributed training provided by the embodiments of the present disclosure can be applied to technical fields such as artificial intelligence, machine learning, deep learning, and distributed training, which are not limited herein. The data parallel communication method in distributed training provided by the embodiments of the present disclosure can optimize data parallel communication during large-scale distributed training, improve the training efficiency of deep learning models, and is applicable to scenarios of multi-parallel strategy combinations (such as data parallelism, tensor parallelism, and pipeline parallelism) in deep learning frameworks. The embodiments of the present disclosure can support the implementation of mainstream deep learning frameworks (such as PyTorch, TensorFlow), and are adapted to GPUs (Graphics Processing Units), TPUs (Tensor Processing Units), and high-performance distributed computing environments.

[0114] The following uses a specific application scenario to illustrate the data parallel communication method in distributed training provided by the embodiments of the present disclosure. In this application scenario, the target model can adopt the Transformer network structure, and the typical modules of the Transformer network perform forward propagation in the following order: query-key-value linear transformation module (qkvlinear) → FlashAttention module → downsampling projection module (downproj) → first fully-connected layer (fc1) → second fully-connected layer (fc2).

[0115] In this application scenario, the reduction and scattering operation of the parameter gradient dW of the second fully-connected layer (fc2) can be completed during the calculation of the input gradient of the dimensionality reduction projection module (downproj); the reduction and scattering operation of the parameter gradients dW of the first fully-connected layer (fc1) and the dimensionality reduction projection module (downproj) can be completed during the backpropagation calculation task of the FlashAttention module; the reduction and scattering operation of the parameter gradient dW of the query-key-value linear transformation module (qkvlinear) other than the first layer can be inserted into the calculation process of the input gradient of the second fully-connected layer (fc2) of the upper-layer Transformer module; the reduction and scattering operation of the parameter gradient dW of the query-key-value linear transformation module (qkvlinear) of the first layer can be completed during the calculation of the embedding vector.

[0116] In this application scenario, the global collection operation of the dimensionality reduction projection module (downproj) can be completed during the forward propagation calculation process of the query-key-value linear transformation module (qkvlinear); the global collection operation of the first fully-connected layer (fc1) can be completed during the forward propagation calculation process of the dimensionality reduction projection module (downproj); the global collection operation of the second fully-connected layer (fc2) can be completed during the forward propagation calculation process of the first fully-connected layer (fc1); the global collection operation of the query-key-value linear transformation module (qkvlinear) of the next layer can be completed during the forward propagation calculation process of the second fully-connected layer (fc2).

[0117] In the training of large-scale deep learning models, training strategies usually combine multiple parallel methods such as tensor parallelism, pipeline parallelism, and data parallelism to form a comprehensive parallel strategy. In such a complex parallel scenario, the time allocation and dependency relationship between communication tasks and computing tasks are crucial. Embodiments of the present disclosure optimize the execution timing of communication operators, reasonably arrange reduction and scattering operations and global collection operations, so that communication time is fully hidden in computing time, thereby being able to accelerate the training iteration speed, shorten the model convergence time, and significantly improve training efficiency. In addition, embodiments of the present disclosure can exploit the potential of computing devices under existing hardware conditions, improve the cost performance of unit hardware resources, and reduce the overall operating cost of the computing cluster. Embodiments of the present disclosure are applicable to combined scenarios of multiple parallel strategies such as tensor parallelism, pipeline parallelism, and data parallelism, and flexibly coordinate communication requirements in different modes.

[0118] It can be understood that, without violating the principle logic, the above-mentioned method embodiments mentioned in the present disclosure can be combined with each other to form combined embodiments. For the sake of brevity, the present disclosure will not elaborate further. Those skilled in the art can understand that in the above-mentioned methods of the specific implementation manner, the specific execution order of each step should be determined according to its function and possible internal logic.

[0119] In addition, the present disclosure also provides a data parallel communication device, a non-volatile computer-readable storage medium, and a computer program product in distributed training. The above can all be used to implement any data parallel communication method provided by the present disclosure. The corresponding technical solutions and technical effects can be seen in the corresponding records in the method part and will not be elaborated further.

[0120] Figure 2 The block diagram of the data parallel communication device in distributed training provided by the embodiments of the present disclosure is shown. As Figure 2 shown, the data parallel communication device in distributed training includes:

[0121] A forward propagation processing module 21, configured to, for any computing device in a computing device cluster participating in the distributed training of a target model, obtain the globally latest parameters of the target model through a global collection operation during the forward propagation process, and perform forward propagation calculation based on the training data subset corresponding to the computing device and the globally latest parameters of the target model to obtain the loss value corresponding to the computing device, where different computing devices in the computing device cluster perform forward propagation calculation based on different training data subsets;

[0122] A backward propagation processing module 22, configured to, for any of the computing devices, calculate the gradient corresponding to the computing device based on the loss value corresponding to the computing device during the backward propagation process, and transmit the gradient corresponding to the computing device to other computing devices in the computing device cluster through a reduction scatter operation.

[0123] In a possible implementation manner, the backward propagation processing module 22 is configured to:

[0124] Transmit the gradient corresponding to the computing device to other partial computing devices in the computing device cluster through a reduction scatter operation.

[0125] In a possible implementation manner, the forward propagation processing module 21 is configured to:

[0126] For any computing device in a computing device cluster participating in the distributed training of a target model, during the forward propagation process, obtain the locally latest parameters of the target model from each of the other computing devices in the computing device cluster through a global collection operation;

[0127] Determine the global up-to-date parameters of the target model based on the local up-to-date parameters of the target model corresponding to the computing device and the local up-to-date parameters of the target model corresponding to each of the other computing devices.

[0128] In a possible implementation,

[0129] The backpropagation processing module 22 is configured to: for a first module of the target model, in response to the first module not being the first module of the target model, during the backpropagation calculation of a second module, transmit the gradient of the first module corresponding to the computing device to other computing devices in the computing device cluster through a reduce-scatter operation, where the first module performs backpropagation calculation earlier than the second module;

[0130] Alternatively, the forward propagation processing module 21 is configured to: for the first module, in response to the first module being the first module of the target model, during the forward propagation of the target model, transmit the gradient of the first module corresponding to the computing device to other computing devices in the computing device cluster through a reduce-scatter operation.

[0131] In a possible implementation, the target model adopts a Transformer network structure, and the modules in the target model include a query-key-value linear transformation module, a fast attention module, a dimensionality reduction projection module, a first fully connected layer, and a second fully connected layer.

[0132] In a possible implementation, the backpropagation processing module 22 is used for at least one of the following:

[0133] During the calculation of the input gradient of the dimensionality reduction projection module, transmit the parameter gradient of the second fully connected layer corresponding to the computing device to other computing devices in the computing device cluster through a reduce-scatter operation;

[0134] During the backpropagation calculation of the fast attention module, transmit the parameter gradient of the first fully connected layer and the parameter gradient of the dimensionality reduction projection module corresponding to the computing device to other computing devices in the computing device cluster through a reduce-scatter operation;

[0135] For the query-key-value linear transformation module that is not in the first layer, during the calculation of the input gradient of the second fully connected layer in the previous layer, transmit the parameter gradient of the query-key-value linear transformation module corresponding to the computing device to other computing devices in the computing device cluster through a reduce-scatter operation.

[0136] In a possible implementation, the forward propagation processing module 21 is configured to:

[0137] For the query-key-value linear transformation module of the first layer, during the calculation of the embedding vector, the parameter gradients of the query-key-value linear transformation module corresponding to the computing device are transmitted to other computing devices in the computing device cluster through a reduction-scatter operation.

[0138] In a possible implementation, the forward propagation processing module 21 is used to:

[0139] During the forward propagation calculation of the third module of the target model, the globally latest parameters of the fourth module of the target model are obtained through a global collection operation, where the fourth module is the next module of the third module.

[0140] In some embodiments, the functions or modules included in the device provided in the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments. The specific implementation and technical effects can refer to the descriptions of the above method embodiments. For the sake of brevity, they will not be repeated here.

[0141] The embodiments of the present disclosure further provide a data parallel communication device in distributed training, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the steps of the above method.

[0142] The embodiments of the present disclosure further provide a non-volatile computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0143] The embodiments of the present disclosure further provide a computer program product, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program. When the computer program is executed by a processor, the steps of the above method are implemented.

[0144] Figure 3 It is a block diagram of a data parallel communication device 1900 in distributed training shown according to an exemplary embodiment. For example, the device 1900 can be provided as a server or a terminal device. Referring to Figure 3 , the device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by a memory 1932 for storing instructions executable by the processing component 1922, such as application programs. The application programs stored in the memory 1932 can include one or more modules each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to execute the above method.

[0145] The apparatus 1900 may also include a power supply component 1926 configured to perform power management of the apparatus 1900, a wired or wireless network interface 1950 configured to connect the apparatus 1900 to a network, and an input / output interface 1958 (I / O interface). The apparatus 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0146] In an exemplary embodiment, a non-transitory computer-readable storage medium is also provided, such as the memory 1932 including computer program instructions, and the computer program instructions can be executed by the processing component 1922 of the apparatus 1900 to complete the above method.

[0147] A computer-readable storage medium may be a tangible device that can hold and store programs / instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punched card or raised structures in grooves storing instructions thereon, and any suitable combination of the above. The computer-readable storage medium used herein is not construed as an instantaneous signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through an optical fiber cable), or electrical signals transmitted through wires.

[0148] The computer programs (or computer-readable program instructions) described herein can be downloaded from the computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0149] A computer program (or computer program instructions) for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer-readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present disclosure.

[0150] Aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0151] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, create a means for implementing the functions / acts specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which causes a computer, a programmable data processing apparatus, and / or other devices to operate in a specific manner, such that the computer-readable medium storing the instructions includes a manufacture, which includes instructions for implementing various aspects of the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0152] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, causing a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0153] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a segment of a program, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the boxes may occur in a different order than noted in the figures. For example, two consecutive boxes may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each box of the block diagrams and / or flowcharts, and combinations of boxes in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions.

[0154] The computer program product may be implemented specifically in the form of hardware, software, or a combination thereof. In an alternative embodiment, the computer program product is specifically embodied as a computer storage medium. In another alternative embodiment, the computer program product is specifically embodied as a software product, such as a Software Development Kit (SDK), etc.

[0155] The above descriptions of the various embodiments tend to emphasize the differences between the various embodiments. Their similarities or resemblances can be referred to each other. For the sake of brevity, they will not be elaborated herein.

[0156] If the technical solution of an embodiment of the present disclosure involves personal information, before the product applying the technical solution of the embodiment of the present disclosure processes personal information, it has clearly informed the personal information processing rules and obtained the individual's independent consent. If the technical solution of an embodiment of the present disclosure involves sensitive personal information, before the product applying the technical solution of the embodiment of the present disclosure processes sensitive personal information, it has obtained the individual's separate consent and at the same time meets the requirements of "express consent". For example, at a personal information collection device such as a camera, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If an individual voluntarily enters the collection scope, it is regarded as consenting to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are informed by obvious signs / information, personal authorization is obtained through pop-up messages or by asking the individual to upload their personal information by themselves; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0157] The embodiments of the present disclosure have been described above. The above description is exemplary and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art in the technical field without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or the improvement of the technology in the market, or to enable other ordinary skill in the art in the technical field to understand the embodiments disclosed herein.

Claims

1. A data parallel communication method in distributed training, characterized in that Including: For any computing device in a cluster of computing devices participating in the distributed training of a target model, during the forward propagation process, obtain the globally latest parameters of the target model through a global collection operation, and perform forward propagation calculation based on the training data subset corresponding to the computing device and the globally latest parameters of the target model to obtain the loss value corresponding to the computing device, where different computing devices in the cluster of computing devices perform forward propagation calculation based on different training data subsets; For any of the computing devices, during the backpropagation process, calculate the gradient corresponding to the computing device based on the loss value corresponding to the computing device, and transmit the gradient corresponding to the computing device to other computing devices in the cluster of computing devices through a reduction scatter operation.

2. The method according to claim 1, characterized in that, The transmitting the gradient corresponding to the computing device to other computing devices in the cluster of computing devices through a reduction scatter operation includes: Transmitting the gradient corresponding to the computing device to other partial computing devices in the cluster of computing devices through a reduction scatter operation.

3. The method according to claim 1, characterized in that, The obtaining the globally latest parameters of the target model through a global collection operation for any computing device in the cluster of computing devices participating in the distributed training of the target model during the forward propagation process includes: For any computing device in the cluster of computing devices participating in the distributed training of the target model, during the forward propagation process, respectively obtain the locally latest parameters of the target model from other respective computing devices in the cluster of computing devices through a global collection operation; Determine the globally latest parameters of the target model according to the locally latest parameters of the target model corresponding to the computing device and the locally latest parameters of the target model corresponding to other respective computing devices.

4. The method according to any one of claims 1 to 3, characterized in that, The transmitting the gradient corresponding to the computing device to other computing devices in the cluster of computing devices through a reduction scatter operation includes: For a first module of the target model, in response to the first module not being the first module of the target model, during the backpropagation calculation process of a second module, transmit the gradient of the first module corresponding to the computing device to other computing devices in the cluster of computing devices through a reduction scatter operation, where the first module performs backpropagation calculation earlier than the second module; Alternatively, for the first module, in response to the first module being the first module of the target model, during the forward propagation process of the target model, transmit the gradient of the first module corresponding to the computing device to other computing devices in the cluster of computing devices through a reduction scatter operation.

5. The method according to claim 4, characterized in that, The target model adopts a Transformer network structure, and the modules in the target model include a query-key-value linear transformation module, a fast attention module, a dimensionality reduction projection module, a first fully connected layer, and a second fully connected layer.

6. The method according to claim 5, wherein For the first module of the target model, in response to the first module not being the first module of the target model, during the backpropagation calculation of the second module, the gradients of the first module corresponding to the computing device are transmitted to other computing devices in the computing device cluster through a reduce-scatter operation, where the first module performs backpropagation calculation earlier than the second module, including at least one of the following: During the calculation of the input gradients of the dimensionality reduction projection module, the parameter gradients of the second fully connected layer corresponding to the computing device are transmitted to other computing devices in the computing device cluster through a reduce-scatter operation; During the backpropagation calculation of the fast attention module, the parameter gradients of the first fully connected layer and the dimensionality reduction projection module corresponding to the computing device are transmitted to other computing devices in the computing device cluster through a reduce-scatter operation; For the query-key-value linear transformation module that is not in the first layer, during the calculation of the input gradients of the second fully connected layer in the previous layer, the parameter gradients of the query-key-value linear transformation module corresponding to the computing device are transmitted to other computing devices in the computing device cluster through a reduce-scatter operation.

7. The method according to claim 5, wherein For the first module, in response to the first module being the first module of the target model, during the forward propagation of the target model, the gradients of the first module corresponding to the computing device are transmitted to other computing devices in the computing device cluster through a reduce-scatter operation, including: For the query-key-value linear transformation module in the first layer, during the calculation of the embedding vector, the parameter gradients of the query-key-value linear transformation module corresponding to the computing device are transmitted to other computing devices in the computing device cluster through a reduce-scatter operation.

8. The method according to any one of claims 1 to 3, characterized in that, Obtaining the globally latest parameters of the target model through a global gather operation, including: During the forward propagation calculation of the third module of the target model, obtaining the globally latest parameters of the fourth module of the target model through a global gather operation, where the fourth module is the next module of the third module.

9. A data parallel communication device in distributed training, characterized in that, Including: A forward propagation processing module, configured to, for any computing device in the computing device cluster participating in the distributed training of the target model, during the forward propagation, obtain the globally latest parameters of the target model through a global gather operation, and perform forward propagation calculation based on the training data subset corresponding to the computing device and the globally latest parameters of the target model to obtain the loss value corresponding to the computing device, where different computing devices in the computing device cluster perform forward propagation calculation based on different training data subsets; A backpropagation processing module, configured to, for the any computing device, during the backpropagation, calculate the gradients corresponding to the computing device based on the loss value corresponding to the computing device, and transmit the gradients corresponding to the computing device to other computing devices in the computing device cluster through a reduce-scatter operation.

10. A data parallel communication device in distributed training, comprising a memory, a processor, and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

11. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, or a non-volatile computer-readable storage medium carrying the computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images