Distributed training method and system, electronic device and storage medium

Abstract

The embodiments of the present disclosure provide a distributed training method and system, an electronic device, and a storage medium. The distributed training method is applied to a distributed training system. In the current round of iterative training, the training of at least one stage of the target model is collaboratively performed by at least one corresponding device and at least one corresponding host. The distributed training method includes: for each stage of the target model, the device performs a forward calculation task to obtain a forward calculation result; the device performs a reverse calculation task based on the forward calculation result to obtain gradient data; in response to the completion of the reverse calculation task, the device triggers the execution of an optimization update task and obtains updated parameters for use in the next round of iterative training. The distributed training method can reduce waiting time, save computing resources of the device, achieve efficient overlap of communication and computing, effectively improve pipeline efficiency and parallelism, and thus alleviate the communication bottleneck problem in large-scale distributed deep learning training.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a distributed training method and system, an electronic device, and a storage medium. Background Art

[0002] With the development of technology, artificial intelligence (AI) has been widely applied in various fields. Deep learning is one of the key technologies in AI. Deep learning technology based on artificial neural networks has made great progress in fields such as object classification, text processing, image search, and human-computer interaction.

[0003] As the complexity of problems increases, the depth and scale of deep learning models continue to improve, and distributed training is widely used for large-scale model training. Summary of the Invention

[0004] At least one embodiment of the present disclosure provides a distributed training method, which is applied to a distributed training system, wherein the distributed training system is configured to perform distributed training on a target model, and the distributed training system includes multiple computing nodes, each computing node includes at least one host and at least one device, wherein the target model is divided into multiple stages in a pipeline parallel mode, each stage includes at least one layer of the target model, and the multiple stages are respectively distributed in multiple devices of the multiple computing nodes, wherein, in a current round of iterative training, the training of at least one stage of the target model is collaboratively performed by at least one corresponding device and at least one corresponding host.

[0005] The distributed training method includes: for each stage of the target model, the device performs a forward calculation task to obtain a forward calculation result; the device performs a reverse calculation task based on the forward calculation result to obtain gradient data; in response to the completion of the reverse calculation task, the device triggers the execution of an optimization update task and obtains updated parameters for use in the next round of iterative training; for at least one stage of the target model, in response to the completion of the reverse calculation task, the device triggers the execution of an optimization update task and obtains updated parameters for use in the next round of iterative training, including: in response to the completion of the reverse calculation task, the device transmits at least part of the gradient data to the host and triggers the host to execute the optimization update task; the host executes the optimization update task based on at least part of the gradient data, obtains at least part of the updated parameters, and transmits the at least part of the updated parameters to the device for use in the next round of iterative training.

[0006] In the distributed training method provided by at least one embodiment of the present disclosure, the devices in the multiple computing nodes are divided into multiple data parallel groups in a data parallel mode, gradient aggregation is performed between devices in the same data parallel group, and in response to the completion of the reverse computing task, the device transmits at least part of the gradient data to the host, including: for each data parallel group related to the stage, in response to the completion of the reverse computing task, each device in the data parallel group transmits the gradient data to the corresponding host through device-to-host transmission.

[0007] In the distributed training method provided in at least one embodiment of the present disclosure, the host performs an optimization update task based on at least part of the gradient data, obtains at least part of the updated parameters, and passes the at least part of the updated parameters to the device for use in the next round of iterative training, including: in each data parallel group related to the stage, for the host corresponding to each device in the data parallel group, the host divides the gradient data passed by the device into N parts of data, and sends N-1 parts of the N parts of data to the hosts corresponding to other devices in the data parallel group, respectively, where N is the number of devices in the data parallel group; the host performs an optimization update task based on all the received gradient data to obtain at least part of the updated parameters; the host obtains the updated parameters generated by the hosts corresponding to other devices in the data parallel group, and passes all the updated parameters to the devices through host-to-device transmission for use in the next round of iterative training.

[0008] In the distributed training method provided by at least one embodiment of the present disclosure, the devices in the multiple computing nodes are divided into multiple data parallel groups in a data parallel mode, and gradient aggregation is performed between devices in the same data parallel group. In response to the completion of the reverse computing task, the device transfers at least part of the gradient data to the host, including: for each data parallel group related to the stage, in response to the completion of the reverse computing task, each device in the data parallel group divides the gradient data into N parts of data, transfers one part of the N parts of data to the corresponding host through device-to-host transmission, and sends the remaining N-1 parts of the N parts of data to the hosts corresponding to other devices in the data parallel group, respectively, where N is the number of devices in the data parallel group.

[0009] In the distributed training method provided in at least one embodiment of the present disclosure, the host performs an optimization update task based on at least part of the gradient data, obtains at least part of the updated parameters, and transmits the at least part of the updated parameters to the device for use in the next round of iterative training, including: in each data parallel group related to the stage, for the host corresponding to each device in the data parallel group, the host performs an optimization update task based on all received gradient data, obtains at least part of the updated parameters, and sends the at least part of the updated parameters to all devices in the data parallel group for use in the next round of iterative training.

[0010] The distributed training method provided by at least one embodiment of the present disclosure also includes: allocating the i-th stage and the M-i+1-th stage to the same computing node, where 0<i≤M / 2, M is the total number of stages obtained by dividing the target model, and the i-th stage represents the i-th stage executed sequentially during the forward propagation process of the target model.

[0011] In the distributed training method provided by at least one embodiment of the present disclosure, the at least one stage includes a first stage, and the first stage includes the front-most layer in the target model in the forward propagation order.

[0012] In the distributed training method provided in at least one embodiment of the present disclosure, the configuration of the host corresponding to the first stage is higher than the configuration of the host corresponding to other stages.

[0013] In the distributed training method provided in at least one embodiment of the present disclosure, each computing node further includes at least one network card, and the at least one host is configured to drive the at least one network card to achieve network communication among the multiple computing nodes.

[0014] In the distributed training method provided by at least one embodiment of the present disclosure, the network communication is based on a remote direct memory access mechanism.

[0015] At least one embodiment of the present disclosure provides a distributed training system, which is configured to perform distributed training on a target model, wherein the distributed training system includes: multiple computing nodes, each computing node includes at least one host and at least one device, wherein the target model is divided into multiple stages in a pipeline parallel mode, each stage includes at least one layer of the target model, and the multiple stages are respectively distributed in multiple devices of the multiple computing nodes; wherein, in the current round of iterative training, the training of at least one stage of the target model is collaboratively performed by at least one corresponding device and at least one corresponding host, and for each stage of the target model, the device is configured to : Execute a forward calculation task to obtain a forward calculation result; execute a reverse calculation task based on the forward calculation result to obtain gradient data; in response to the completion of the reverse calculation task, trigger the execution of an optimization update task and obtain updated parameters for use in the next round of iterative training; for at least one stage of the target model, the device is further configured to: in response to the completion of the reverse calculation task, transfer at least part of the gradient data to the host, and trigger the host to execute the optimization update task; the host is configured to: execute the optimization update task based on at least part of the gradient data, obtain at least part of the updated parameters, and transfer the at least part of the updated parameters to the device for use in the next round of iterative training.

[0016] At least one embodiment of the present disclosure provides an electronic device, comprising: at least one processor; at least one memory, comprising one or more computer program modules; wherein the one or more computer program modules are stored in the at least one memory and configured to be executed by the at least one processor, and the one or more computer program modules are used to implement the distributed training method provided by at least one embodiment of the present disclosure.

[0017] At least one embodiment of the present disclosure provides a non-transitory computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions, when executed by at least one processor, perform the distributed training method provided by at least one embodiment of the present disclosure.

[0018] The distributed training method and system, electronic device, and storage medium provided in at least one embodiment of the present disclosure can reduce waiting time and effectively improve pipeline efficiency and parallelism by immediately triggering the execution of an optimization and update task upon the completion of a reverse computation task. Furthermore, by migrating the optimization and update task from the device to the host, which then completes the optimization and update task, the device's computing resources can be conserved, achieving efficient overlap between communication and computation, thereby alleviating the communication bottleneck problem in large-scale distributed deep learning training. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, rather than limiting the present disclosure.

[0020] Figure 1A A schematic diagram of a data parallel mode;

[0021] Figure 1B A schematic diagram of a tensor parallel mode;

[0022] Figure 1C A schematic diagram of a pipeline parallel mode;

[0023] Figure 1D A schematic diagram of a micro-batch pipeline parallel mode;

[0024] Figure 2A A schematic block diagram of a distributed training system provided in at least one embodiment of the present disclosure;

[0025] Figure 2B A schematic block diagram of a computing node provided in at least one embodiment of the present disclosure;

[0026] Figure 3A A flowchart of a distributed training method provided in at least one embodiment of the present disclosure;

[0027] Figure 3B A timing diagram of a distributed training method provided in at least one embodiment of the present disclosure;

[0028] Figure 3C A flowchart of a distributed training method provided in at least one embodiment of the present disclosure;

[0029] Figure 4 An exemplary schematic diagram of a distributed training method provided for at least one embodiment of the present disclosure;

[0030] Figure 5 An exemplary schematic diagram of a distributed training method provided for at least one embodiment of the present disclosure;

[0031] Figure 6 A schematic diagram of a return pipeline provided by at least one embodiment of the present disclosure;

[0032] Figure 7 A schematic block diagram of an electronic device provided in at least one embodiment of the present disclosure;

[0033] Figure 8 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure;

[0034] Figure 9A schematic block diagram of a non-transitory computer-readable storage medium is provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0036] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by persons of ordinary skill in the field to which this disclosure belongs. The words "first", "second" and similar terms used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0037] The present disclosure is described below using several specific embodiments. To maintain clarity and conciseness in the following description of the embodiments of the present disclosure, detailed descriptions of known functions and components may be omitted. When any component of an embodiment of the present disclosure appears in more than one drawing, that component is represented by the same or similar reference numeral in each drawing.

[0038] A wide variety of processors are used for model training, including graphics processing units (GPUs), general-purpose graphics processing units (GPGPUs), tensor processing units (TPUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and artificial intelligence (AI) accelerators or coprocessors. Currently, GPUs are typically used as the primary training device, combined with central processing units (CPUs) for auxiliary tasks and overall process management. For specific scenarios or requirements, customized acceleration may be achieved using ASICs, FPGAs, or AI accelerators.

[0039] Because a large-scale model can have hundreds of billions of parameters, a single machine or GPU is no longer sufficient for training such models. Therefore, distributed training is widely used for large-scale model training in artificial intelligence. Distributed training utilizes multiple machines to collaborate to accelerate the training of large deep learning models. In distributed training, the original model, dataset, and training process are broken down and distributed across multiple machines for simultaneous processing, effectively utilizing more computing resources and shortening training time.

[0040] In the distributed training of large models, common parallel technologies include data parallelism and model parallelism. Model parallelism further includes tensor parallelism and pipeline parallelism.

[0041] The data parallel mode refers to splitting the model's training dataset into multiple sub-datasets. The divided sub-datasets are distributed to multiple devices. Each device holds a complete copy of the model, so that the sub-datasets assigned to it can be trained independently. For example, each device can independently complete the forward propagation and backward propagation calculations of the sub-dataset.

[0042] Figure 1A This is a schematic diagram of a data parallel mode. Figure 1A As shown in the figure, a training dataset is split into three parts and distributed to device 0, device 1, and device 2, respectively. Each device 0, device 1, and device 2 has a complete model. After backpropagation is completed on each device, the gradients calculated by each device need to be aggregated and the global model parameters updated.

[0043] Figure 1B This is a diagram of a tensor parallel mode. The tensor parallel mode is to split a layer (or operator) of the model and place some weights of the operator on different devices, thereby reducing the memory usage on each device (for example, GPU). A neural network model consists of multiple layers ( Figure 1B ), each layer can be understood as a function that can perform specific mathematical operations on the input tensor. For example, the fully connected layer can perform linear transformation operations on the tensor, the convolutional layer can perform convolution operations on the tensor, and the pooling layer can perform pooling operations on the tensor to extract or transform features. These operations can change the content and shape of the tensor, realizing the gradual transformation from raw data to high-level abstract representations. Figure 1B As shown in the figure, in the tensor parallel mode, the tensor of a layer is divided into multiple parts, and the divided tensors are respectively distributed to multiple devices, such as device 0, device 1 and device 2. Each device is only responsible for part of the model calculation and exchanges necessary intermediate results through communication.

[0044] Pipeline parallelism divides the different layers of a model into multiple stages, each of which is assigned to multiple devices to form a pipeline. Activation values and gradients are passed sequentially between devices. For example, during forward propagation, each device passes activation values to the device in the next pipeline stage. During backward propagation, each device passes gradients back to the device in the previous pipeline stage. This allows computations to be performed on multiple devices simultaneously, improving computational efficiency.

[0045] Figure 1C This is a schematic diagram of a pipeline parallel mode. Figure 1C As shown, a model includes 4 layers, which are divided into 3 stages. Each stage is assigned to a device. For example, the first layer is assigned to device 0, the second layer is assigned to device 1, and the third and fourth layers are assigned to device 2.

[0046] Figure 1D This diagram illustrates a micro-batch pipeline parallelism model. By breaking global batch data into micro-batches and artificially creating pipelines to address device idleness, allowing different devices to participate in the computation simultaneously, this significantly improves pipeline parallel device utilization and reduces device idle time.

[0047] like Figure 1D As shown, the multiple layers of the target model are split into 4 stages, each stage includes, for example, 4 micro-batches of data, F i,j represents the forward propagation of the jth micro-batch data in the i-th stage, B x,yrepresents the back propagation of the yth micro-batch data in the xth stage. In the forward calculation process, the calculation of the later stage depends on the calculation of the previous stage, for example, F 1,0 Must be in F 0,0 The calculation can only be started after the calculation is completed. Similarly, in the reverse calculation process, the calculation of the previous stage needs to depend on the calculation of the next stage, for example, B 2,3 Must be in B 3,3 Calculation can only start after the calculation is completed.

[0048] In pipeline parallel mode, data is transmitted between adjacent devices via communication links. For example, during the forward computation, the input data is first computed on device 0 through the first layer to obtain an intermediate result. This intermediate result is then transmitted to device 1. The second layer's output is then computed on device 1, and the model's second layer output is transmitted to device 2. The final result of the forward computation is then computed on device 2 through the final layer. The backward propagation process is similar. Finally, the network layers on each device update their parameters using the gradients calculated during the backward propagation process.

[0049] exist Figure 1D In the pipeline parallel mode shown, reverse computation can only begin after all devices have completed forward computation. This means that the intermediate results of the first stage's forward computation must be retained until all stages of forward computation have completed. This results in a large number of pipeline bubbles—idle cycles or gaps in the pipeline. Excessive bubbles can degrade processor performance, so reducing them is a key goal in optimizing processor performance.

[0050] As models scale, using a single parallel mode often cannot simultaneously meet device memory limitations and high computational efficiency requirements. Therefore, for large-scale models, it is often necessary to combine multiple parallel technologies, such as data parallelism, tensor parallelism, and pipeline parallelism, for distributed training.

[0051] However, the inventors of the present disclosure have noticed that in a training scenario using large-scale distributed parallelism, communication overhead becomes a bottleneck that restricts the overall training performance. In the data parallel mode, as the scale of deep learning models increases, communication overhead becomes a key factor limiting performance improvement. For example, the proportion of data parallel communication is high, making it difficult to fully utilize network bandwidth, resulting in a low transmission rate. In addition, as the scale of data parallelism expands, the bandwidth of the aggregate communication algorithm decreases significantly. Although some methods based on streaming multiprocessors can increase bandwidth, they cannot achieve overlap between communication and computing. Other methods use CPU optimizers to offload parameters to the CPU. Although this can reduce the burden on the GPU to a certain extent, the communication process still needs to be completed by the GPU, and the parallelization of backpropagation and communication cannot be achieved.

[0052] At least one embodiment of the present disclosure provides a distributed training method, which is applied to a distributed training system, wherein the distributed training system is configured to perform distributed training on a target model, and the distributed training system includes multiple computing nodes, each computing node includes at least one host and at least one device, wherein the target model is divided into multiple stages in a pipeline parallel mode, each stage includes at least one layer of the target model, and the multiple stages are respectively distributed in multiple devices of the multiple computing nodes, wherein, in the current round of iterative training, the training of at least one stage of the target model is collaboratively performed by at least one corresponding device and at least one corresponding host.

[0053] The distributed training method includes: for each stage of the target model, the device performs a forward calculation task to obtain a forward calculation result; the device performs a reverse calculation task based on the forward calculation result to obtain gradient data; in response to the completion of the reverse calculation task, the device triggers the execution of an optimization update task and obtains the updated parameters for use in the next round of iterative training.

[0054] For at least one stage of the target model, in response to the completion of the reverse calculation task, the device triggers the execution of the optimization update task and obtains the updated parameters for use in the next round of iterative training, including: in response to the completion of the reverse calculation task, the device transmits at least part of the gradient data to the host and triggers the host to execute the optimization update task; the host executes the optimization update task based on at least part of the gradient data, obtains at least part of the updated parameters, and transmits at least part of the updated parameters to the device for use in the next round of iterative training.

[0055] The distributed training method provided by at least one embodiment of the present disclosure reduces waiting time and effectively improves pipeline efficiency and parallelism by immediately triggering the execution of an optimization and update task upon the completion of a reverse computation task. Furthermore, by migrating the optimization and update task from the device to the host, which then completes the optimization and update task, it conserves computing resources on the device and enables efficient overlap of communication and computation, thereby alleviating the communication bottleneck problem in large-scale distributed deep learning training.

[0056] The distributed training method provided in at least one embodiment of the present disclosure can be applied to a distributed training system. In an embodiment of the present disclosure, the distributed training system is configured to perform distributed training on a target model. The distributed training system includes multiple computing nodes, each of which includes at least one host and at least one device.

[0057] Figure 2A This is a schematic block diagram of a distributed training system provided by at least one embodiment of the present disclosure. For example, Figure 2A The distributed training system shown includes multiple compute nodes (compute node 0 through compute node x). Each compute node includes multiple hosts (host 0 through host y). Each host is physically connected to multiple devices (device 0 through device z) (e.g., via physical lines such as Ethernet cables and optical fibers, as well as network hardware such as switches). x, y, and z are all positive integers and can be set based on actual needs. Within each compute node, data can be transmitted between the host and the devices in various ways. Devices (also called "slaves") can either connect directly to the host through a hardware interface (e.g., via the Peripheral Component Interconnect Express (PCIe) bus) to achieve low-latency data transmission, such as device-to-host (D2H) and host-to-device (H2D) transfers. Alternatively, they can collaborate with the host over the network in a software-defined manner, such as through virtualization technology. It should be noted that different compute nodes have different hosts, and the devices corresponding to different hosts are also different. Specifically, host 0 in compute node 0 is not the same host as host 0 in compute node x, and device 0 connected to host 0 is not the same device as device 0 connected to host 1, and so on.

[0058] It should be noted that Figure 2A As an example, in the above distributed training system, each computing node may include only one host, and each host may be physically connected to only one device. Furthermore, the number of hosts in each computing node may vary, and the number of devices physically connected to each host may also vary.

[0059] For example, in at least one example of the embodiments of the present disclosure, the host may include a central processing unit (CPU), and the device may include a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an AI accelerator, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other accelerators. For example, a distributed training system may include 8, 16, or 32 computing nodes, each of which may include 4 or 8 GPUs.

[0060] In an embodiment of the present disclosure, each computing node may further include at least one network card, and at least one host is configured to drive one network card to implement network communication among multiple computing nodes.

[0061] Figure 2B A schematic block diagram of a computing node provided in at least one embodiment of the present disclosure.

[0062] For example, Figure 2B As shown in the figure, the computing node includes host 0~host 1, switch 0~switch 3, device 0~device 7 and network card 0~network card 7. For example, host 0, device 0 and device 1, network card 0 and network card 1 are connected through switch 0. Host 0 can drive network card 0~network card 3 to realize communication between the computing node and other computing nodes in the distributed training system. For network cards and devices connected through the same switch, the network card can be called the affinity network card of the device. Figure 2B In the example, NIC 0 and NIC 1 are the affinity NICs for device 0 and device 1. The switch can be, for example, a PCIe switch. Within the same compute node, hosts can be connected via an Ultra Path Interconnect (UPI) bus, and devices can be connected via a high-speed bus or other GPU interconnect technology (not shown).

[0063] It should be noted that Figure 2B This is merely an example. Depending on actual needs, a host can be connected to more or fewer switches, and each switch can be connected to more or fewer devices and network cards. The embodiments of the present disclosure are not limited in this regard. Furthermore, the number of devices and network cards connected to each switch can be unequal. For example, in some examples, a switch can be connected to two devices and one network card. This means that both devices communicate with other computing nodes through the same network card. This network card can be referred to as an affinity network card for these two devices.

[0064] In the distributed training system provided by at least one embodiment of the present disclosure, communication between computing nodes is achieved by using a host-driven network interface card (NIC) to fully utilize the host's multi-threaded processing capabilities, effectively utilizing network bandwidth and improving communication efficiency. This architecture allows devices to focus on executing computing tasks while effectively addressing the issue of insufficient network bandwidth when the device drives the NIC, thereby improving the overall performance of the distributed training system under high loads.

[0065] The distributed training method provided by at least one embodiment of the present disclosure can be applied to various types of distributed training systems, including but not limited to: Figure 2A The distributed training system shown is also not limited to include Figure 2B The distributed training system of computing nodes shown.

[0066] In a distributed training method provided by at least one embodiment of the present disclosure, a target model is divided into multiple stages in a pipeline parallel mode, each stage includes at least one layer of the target model, and the multiple stages are respectively distributed in multiple devices of multiple computing nodes. For example, when a micro-batch pipeline parallel mode is adopted, each stage may include multiple micro-batches. For example, taking the target model including 80 layers as an example, assuming that the target model is divided into 8 stages according to the pipeline parallel mode, each stage includes 10 layers of the target model.

[0067] It should be noted that the multiple stages and multiple computing nodes mentioned above do not need to correspond one to one. A computing node can correspond to one or more stages, and a stage can also correspond to one or more computing nodes, or one or more devices in a computing node.

[0068] For example, the target model mentioned above can be a machine learning model to be trained, such as a deep learning model. A deep learning model can be a neural network structure with multiple layers (3, 4, 8, or more layers), such as a convolutional neural network (CNN), a recurrent neural network (RNN), or a long short-term memory network (LSTM). These neural networks typically include an input layer, hidden layers, and an output layer. Hidden layers refer to the layers between the input and output layers of the neural network and are also called processing layers. The input layer receives data to be processed, such as an image, and the output layer outputs processing results, such as a processed image. Processing layers can include convolutional layers, pooling layers, batch normalization layers, and fully connected layers. Depending on the neural network structure, processing layers can include different content and combinations. In some examples, the target model can be a large language model (LLM), a deep learning model trained on large amounts of text data that can understand and generate natural language text. Large language models are usually based on the Transformer architecture and learn the statistical laws and semantic information of language from massive text data through self-supervised learning. They are usually used for tasks such as text generation, translation, question answering, and summarization.

[0069] Figure 3A A flowchart of a distributed training method provided in at least one embodiment of the present disclosure.

[0070] For example, Figure 3A As shown, in the current round of iterative training, the training of at least one stage of the target model is collaboratively performed by at least one corresponding device and at least one corresponding host. For each stage of the target model, the distributed training method provided by at least one embodiment of the present disclosure includes the following steps S101 to S103.

[0071] Step S101: The device executes a forward calculation task and obtains a forward calculation result.

[0072] Step S102: The device performs a reverse calculation task based on the forward calculation result to obtain gradient data.

[0073] Step S103: In response to the completion of the reverse calculation task, the device triggers the execution of the optimization update task and obtains the updated parameters for use in the next round of iterative training.

[0074] For example, in step S101, forward computation (also known as forward propagation or forward propagation) is a computational method in a neural network used to calculate the relationship between the input layer, hidden layer, and output layer. This involves a series of processes, starting with the input data from the neural network's input layer, through a series of weight matrix and activation function calculations, to ultimately obtain the output result. During forward computation, the neural network uses the output of each layer as the input to the next layer until the output layer reaches the final calculation result.

[0075] Activation values are intermediate results generated by each layer of the network during the forward propagation process. These results serve as the basis for gradient calculation during the backward propagation process. During the forward propagation phase, activation values refer to the result after weighted inputs are applied to the activation function. For a neuron, it first receives signals from neurons in the previous layer (usually a weighted sum plus a bias term). Then, activation values are calculated using a nonlinear activation function (such as Sigmoid or ReLU). The activation value represents the neuron's "level of activation" at a specific moment.

[0076] For example, in step S102, reverse calculation (also known as back propagation) is an optimization algorithm for adjusting weights and biases in a neural network, which can be performed based on, for example, minimizing a loss function. The loss function is a mathematical function used to measure the error between the output of a neural network model and the actual label or target value. During the training of a neural network, for example, back propagation can calculate the gradient of the loss function and propagate the gradient back from the output layer to the input layer, thereby updating the weight parameters of each layer. Weights are parameters that connect neurons in a neural network. Each weight corresponds to the strength of the connection from the neurons in the previous layer to the neurons in the current layer. In forward propagation, weights are used to adjust the contribution of the input signal, affecting the change in the activation value of the neuron output.

[0077] The gradient is the local rate of change of the objective function relative to the model parameters, reflecting the trend in the loss function value as the parameters change. The gradient is used to guide the direction and magnitude of weight updates during the optimization process, allowing the entire network to gradually improve its performance towards reducing prediction error. During the backpropagation phase, the core of the gradient calculation is the partial derivatives of the loss function with respect to each weight and bias. Once the forward propagation completes the network output calculation and calculates the loss function value, the backpropagation process begins, calculating the gradient of the loss function with respect to each layer's weights and biases, layer by layer.

[0078] For example, in step S103, parameters refer to model parameters, including the trainable weights and biases in the neural network. During training, model parameters are not only used multiple times in forward and backward propagation, but also change as the optimizer is updated. Updated parameters refer to the model parameters obtained after the optimization and update task and are used in the next round of iterative training.

[0079] For example, in step S103 , the optimization and updating tasks may include gradient synchronization, parameter optimization, and parameter updating.

[0080] Gradient synchronization refers to the need to aggregate the gradient values calculated by multiple devices in parallel to ensure global gradient consistency in distributed training scenarios. Gradient synchronization is typically achieved through communication operations such as all-reduce, ensuring that the gradient data on multiple devices is consistent.

[0081] Parameter optimization refers to the process of iteratively adjusting model parameters based on gradient data. Essentially, it uses an optimization algorithm to search for the optimal solution in the parameter space that minimizes the loss function. Examples of optimization algorithms include the Stochastic Gradient Descent (SGD) algorithm and the Adaptive Moment Estimation (Adam) optimizer.

[0082] Parameter updating refers to the process of actually adjusting model parameters according to the update rules provided by the optimization algorithm. This means modifying the current model weights and biases according to the algorithm's specifications, gradually approaching the optimal solution. Parameter updating directly impacts model performance and is the process of improving the model after each iteration.

[0083] For example, in step S103, for each stage of the target model, once the reverse calculation task is completed, the device immediately triggers the execution of the optimization update task. It should be noted that the optimization update task can be executed by the device itself, or it can be triggered by the host to execute. For example, in the case of adopting the micro-batch pipeline parallel mode, for each stage of the target model, when the reverse calculation task of the last micro-batch is completed, the device immediately triggers the execution of the optimization update task. Figure 1D It is necessary to wait for all micro-batches in the stage to complete the reverse calculation task (that is, B 0,0 Compared with uniformly executing the optimization and update tasks of each stage after completing the reverse calculation task, the distributed training method proposed in the embodiment of the present disclosure can reduce waiting time and effectively improve pipeline efficiency and parallelism.

[0084] Figure 3B A timing diagram of a distributed training method provided in at least one embodiment of the present disclosure.

[0085] For example, Figure 3BThe figure shows the timing relationship between the reverse computation tasks and the optimization and update tasks in four stages (Stage 0 to Stage 3). Through step S103 above, the optimization and update tasks of Stage 3 overlap in time with the reverse computation tasks of Stage 2, the optimization and update tasks of Stage 2 overlap in time with the reverse computation tasks of Stage 1, and the optimization and update tasks of Stage 1 overlap in time with the reverse computation tasks of Stage 0. This approach effectively hides the communication overhead brought by the optimization and update tasks of Stages 1 to 3, thereby significantly improving resource utilization and overall training efficiency.

[0086] Figure 3C A flowchart of a distributed training method provided in at least one embodiment of the present disclosure.

[0087] For example, Figure 3C As shown, for at least one stage of the target model, an example of step S103 may include the following steps S1031 to S1032.

[0088] Step S1031: In response to the completion of the reverse calculation task, the device transmits at least part of the gradient data to the host, and triggers the host to execute the optimization update task.

[0089] Step S1032: The host performs an optimization update task based on at least part of the gradient data, obtains at least part of the updated parameters, and transmits at least part of the updated parameters to the device for use in the next round of iterative training.

[0090] For example, in steps S1031 and S1032, for the current phase, once the reverse calculation task is completed, the device immediately transmits at least a portion of the gradient data to the host, and simultaneously triggers the host to perform the optimization update task. For example, the device may send a trigger signal to the host to trigger the host to perform the optimization update task. Upon receiving the trigger signal from the device, the host may perform the optimization update task based on at least a portion of the gradient data, obtain at least a portion of the updated parameters, and transmit at least a portion of the updated parameters to the device for use in the next round of iterative training.

[0091] For example, the device can transmit gradient data to the host through D2H transmission or network communication, and the host can transmit updated parameters to the device through H2D transmission or network communication. Specifically, within the same computing node, the device can transmit gradient data to the host through D2H transmission, and the host can transmit updated parameters to the device through H2D transmission. Between different computing nodes, the device can transmit gradient data to the host through network communication, and the host can transmit updated parameters to the device through network communication. For example, network communication can be implemented based on the network card and GPU Direct mechanism. The GPU Direct mechanism allows high-speed data transmission directly from the GPU to the network card. According to actual needs, the device can transmit all gradient data obtained through the reverse calculation task to the host, or only transmit part of the gradient data to the host.

[0092] For example, in data parallel mode, there may be multiple data parallel groups related to the current stage, and gradient aggregation is required between devices within the same data parallel group. Gradient aggregation refers to the process of summarizing the gradient data calculated on multiple devices, which usually involves merging the local gradients on multiple devices into a global gradient. These devices may be located in multiple different computing nodes, and each device has a host physically connected to it. For each device in the same data parallel group, the device can transmit all the gradient data obtained through the reverse calculation task to each host separately (the host here refers to the host physically connected to the above-mentioned devices); the device can also divide all the gradient data into multiple parts, and then transmit each part to different hosts separately.

[0093] For example, when neither data parallelism nor tensor parallelism is used, and there is only one device corresponding to the current stage, this device can transmit all gradient data to the host physically connected to it via D2H transmission, triggering the host to perform an optimization update task. For example, the host can perform an optimization update task based on all the gradient data, obtain updated parameters, and transmit these updated parameters to the device via H2D transmission for use in the next round of training iterations.

[0094] It's important to note that the current stage includes multiple layers. Upon completion of the reverse computation for each layer, the device immediately transmits at least some of the gradient data to the host, triggering the host to perform optimization updates without having to wait for all layers in the current stage to complete their reverse computations. This approach effectively improves pipeline efficiency and parallelism, achieving efficient overlap between communication and computation.

[0095] For example, in the distributed training method provided in at least one embodiment of the present disclosure, the above “at least one stage” may include the first stage, that is, the above steps S1031 to S1032 may be performed only on the first stage. The first stage here refers to the first stage (e.g. Figure 3B Stage 0 in the target model), including the front-end layers in the forward propagation order in the target model.

[0096] Through the above steps S1031 to S1032, the overlap of communication and calculation can be achieved, effectively alleviating the first stage (for example Figure 3B The communication bottleneck problem introduced by the optimization update tasks of Stage 0 in the pipeline timing is not overlapped, which significantly improves the efficiency and performance of the entire pipeline.

[0097] It should be noted that, in addition to the first stage, the above steps S1031 to S1032 may also be performed for other stages of the target model according to actual needs, and the embodiment of the present disclosure does not limit this.

[0098] In the distributed training method provided in the above embodiment of the present disclosure, by migrating optimization and update tasks from the device to the host, the host completes the optimization and update tasks, thereby saving the device's computing resources, effectively improving pipeline efficiency and parallelism, and achieving efficient overlap of communication and computing, thereby alleviating the communication bottleneck problem in large-scale distributed deep learning training. Furthermore, the above method fully utilizes host and memory resources to achieve heterogeneous parallel computing with devices, reducing interference between hardware, and avoiding resource competition and the impact of potential hardware failures.

[0099] In the distributed training method provided by the above embodiment of the present disclosure, the configuration of the host corresponding to the first stage is higher than the configuration of the host corresponding to other stages. For example, the memory capacity of the host corresponding to the first stage can be increased, or the number of processor cores of the host corresponding to the first stage can be increased. In some examples, the memory capacity of the host corresponding to other stages is 1T, the number of processor cores is 128, and the memory capacity of the host corresponding to the first stage is 2T, and the number of processor cores is 256. It should be noted that the above is only an example, and the configuration of each host can be set according to actual needs.

[0100] By improving the configuration of the host corresponding to the first phase, we can effectively support the optimization and update tasks required of that host. This approach only requires optimizing the configuration of the host corresponding to the first phase, providing sufficient computing and memory performance with minimal hardware investment. This approach enables efficient processing of communication and computing tasks during large-scale training, maximizing cost-effectiveness and further alleviating the communication bottleneck problem in large-scale distributed deep learning training.

[0101] It should be noted that if, according to actual needs, the above steps S1031 to S1032 are also performed for certain stages except the first stage, the configuration of the corresponding host can also be improved accordingly. This embodiment of the present disclosure does not limit this.

[0102] In the distributed training method provided by at least one embodiment of the present disclosure, while adopting the pipeline parallel mode, the devices in the multiple computing nodes can also be divided into multiple data parallel groups according to the data parallel mode.

[0103] The following example illustrates a tensor-data-pipeline parallelism model, partitioning in the order of tensor parallelism, data parallelism, and pipeline parallelism. In this example, assuming the target model is partitioned according to TP4-DP2-PP2 (tensor parallelism of 4, data parallelism of 2, and pipeline parallelism of 2), a total of 16 devices (denoted as G0 to G15) are required. A tensor parallelism of 4 means that a single model layer is split into four parts, distributed across four devices for computation; a data parallelism of 2 means that each model replica is trained using two different sets of data, ultimately requiring gradient aggregation to update model parameters; and a pipeline parallelism of 2 means that the entire target model is divided into two stages, executed in a pipelined manner.

[0104] Through the above division operation, 4 tensor parallel groups, 8 data parallel groups and 8 pipeline parallel groups can be obtained.

[0105] The four tensor parallel groups include: [G0, G1, G2, G3], [G4, G5, G6, G7], [G8, G9, G10, G11], and [G12, G13, G14, G15]. The devices in each tensor parallel group work together to complete tensor operations in a model layer.

[0106] The eight data parallel groups include: [G0, G4], [G1, G5], [G2, G6], [G3, G7], [G8, G12], [G9, G13], [G10, G14], and [G11, G15]. Devices within the same data parallel group must perform gradient aggregation. For example, devices G0 and G4 must perform gradient aggregation.

[0107] The eight pipeline parallel groups include: [G0, G8], [G1, G9], [G2, G10], [G3, G11], [G4, G12], [G5, G13], [G6, G14], and [G7, G15]. For example, G0-G7 correspond to the first stage, and G8-G15 correspond to the second stage.

[0108] In the above example, the data parallel groups participating in the first stage of gradient aggregation include [G0, G4], [G1, G5], [G2, G6], and [G3, G7]. The above four data parallel groups can be referred to as data parallel groups related to the first stage.

[0109] It should be noted that the above partitioning method is only an example, and the present disclosure does not limit the partitioning mode, parallelism size, or order. For example, the partitioning can also be performed in the order of tensor parallelism, pipeline parallelism, and data parallelism. The specific partitioning method is similar to the above and is not repeated here.

[0110] In the distributed training method provided in at least one embodiment of the present disclosure, an example of step S1031 may include the following step S201. It should be noted that since step S1031 is performed for "at least one stage" of the target model, the "data parallel group" below should be understood as the data parallel group related to the current stage (i.e., the "at least one stage" targeted by step S1031), and does not involve other stages.

[0111] Step S201: For each data parallel group related to the current stage, in response to the completion of the reverse calculation task, each device in the data parallel group transmits the gradient data to the corresponding host through D2H transmission.

[0112] For example, in step S201, the "corresponding host" refers to a host physically connected to the device, such as a host connected via a switch. For example, in each data parallel group associated with the current stage, the following operations are performed: each device in the data parallel group first completes the forward and reverse calculations locally to obtain gradient data. When the reverse calculation task is completed, each device transmits all generated gradient data to the host physically connected to it via D2H transmission.

[0113] Corresponding to step S201, an example of the above step S1032 may include performing the following operations in each data parallel group related to the current stage: the host corresponding to each device in the data parallel group executes the following steps S202 to S204.

[0114] Step S202: The host divides the gradient data transmitted by the device into N parts of data, and sends N-1 parts of the N parts of data to the hosts corresponding to other devices in the data parallel group, where N is the number of devices in the data parallel group.

[0115] For example, in step S202, each host (referring to the host corresponding to each device in the data parallel group) divides the received gradient data into N copies, retains one copy for itself, and sends the remaining N-1 copies to the hosts corresponding to the other devices in the data parallel group. For ease of description, the host sending data is referred to as the source host, and the host receiving data is referred to as the destination host. If the source host and destination host are not in the same compute node, the source host can send gradient data via network communication. If they are in the same compute node, the source host can send gradient data via an intra-machine interconnect (such as the UPI bus described above).

[0116] Step S203: The host performs an optimization update task based on all received gradient data to obtain at least some updated parameters.

[0117] For example, in step S203, for each host, all received gradient data includes a copy of data retained by itself (i.e., a copy of data not sent in step S202) and data sent by hosts corresponding to other devices in the data parallel group (a total of N-1 copies).

[0118] Step S204: the host obtains updated parameters generated by the host corresponding to other devices in the data parallel group, and transmits all updated parameters to the devices via H2D transmission for use in the next round of iterative training.

[0119] For example, in step S204, if the source and destination hosts are not within the same compute node, the destination host can receive the updated parameters via network communication. If they are within the same compute node, the destination host can receive the updated parameters via an intra-host interconnect (e.g., the UPI bus described above). After each host integrates all received updated parameters, it transmits the integrated parameters back to the device physically connected to the host via H2D transmission for use in the next round of training iterations.

[0120] In the distributed training method provided in the above embodiments of the present disclosure, parameter optimization tasks can be distributed to multiple hosts for collaborative completion through data partitioning, which can effectively improve computing efficiency and reduce data transmission redundancy.

[0121] Figure 4 An exemplary schematic diagram of a distributed training method provided for at least one embodiment of the present disclosure. Figure 4 This is an example of the above steps S201 to S204.

[0122] For example, Figure 4 Shown Figure 3BThe execution process of the reverse calculation task and optimization update task in the first stage (Stage 0) in the example. In this example, Stage 0 includes multiple layers. Figure 4 Only four layers (Layer 0 to Layer 3) are shown in the figure, and the reverse calculation task is performed layer by layer.

[0123] The following describes the execution process of the reverse calculation task and optimization update task of Layer 3. The execution process of Layer 0 to Layer 2 is similar and will not be described here. The following operations are performed for each data parallel group related to Stage 0:

[0124] For example, Figure 4 As shown in Figure 2, when the reverse calculation task of Layer 3 is completed, each device in the data parallel group transmits the gradient data of Layer 3 to the corresponding host through D2H transmission.

[0125] It should be noted that since the D2H transmission time of each layer of gradient data is longer than the execution time of the reverse calculation task of this layer, Figure 4 In the timing diagram shown, "Layer 3 Gradient Data D2H" does not follow "Stage 0 Layer 3" in timing, but rather waits for a period of time. Specifically, the dashed box before "Layer 3 Gradient Data D2H" in the diagram represents the transmission of gradient data D2H from Layers 5 and 4 (not shown). While Layer 4's D2H transmission is not yet complete, Layer 3's reverse calculation has already completed, causing the transmission of Layer 3's gradient data D2H to wait until Layer 4's transmission is complete. This phenomenon is reflected in the timing diagram as a delay before the transmission of Layer 3's gradient data D2H.

[0126] exist Figure 4 In the Reduce-Scatter phase, each host (referring to the host corresponding to each device in the data parallel group) divides the Layer 3 gradient data passed by the device into N parts and sends N-1 parts of the N parts to the hosts corresponding to other devices in the data parallel group, where N is the number of devices in the data parallel group.

[0127] exist Figure 4 In the summation (sum) and optimization (opt) phases, each host performs optimization updates based on all received gradient data to obtain at least some updated parameters of Layer 3. For example, the sum phase corresponds to gradient synchronization, and the opt phase corresponds to parameter optimization.

[0128] exist Figure 4In the all-gather phase, each host obtains the updated parameters generated by the hosts corresponding to other devices in the data parallel group, obtains all updated parameters of Layer 3, and transmits all updated parameters of Layer 3 to the corresponding devices through H2D transmission for use in the next round of iterative training.

[0129] like Figure 4 As shown in the figure, upon completion of each layer's reverse computation, the device immediately transmits the gradient data to the host, triggering the host to execute optimization updates without having to wait for all layers in the current phase to complete their reverse computations. This approach effectively improves pipeline efficiency and parallelism, achieving efficient overlap between communication and computation.

[0130] In the above approach, each device in the data parallel group needs to transmit all of its generated gradient data to the corresponding host via D2H transmission. The host then divides the data and distributes it. To further improve data transmission efficiency, data can be divided and distributed directly at the device, reducing the amount of data transmitted to each host and effectively reducing the time overhead of the data transmission process. Based on this, another example is provided below.

[0131] Another example of step S1031 may include the following step S301. It should also be noted that since step S1031 is performed for "at least one stage" of the target model, the "data parallel group" below should be understood as the data parallel group related to the current stage, and does not involve other stages.

[0132] Step S301: For each data parallel group related to the current stage, in response to the completion of the reverse calculation task, each device in the data parallel group divides the gradient data into N parts of data, transfers one of the N parts of data to the corresponding host through D2H transmission, and sends the remaining N-1 parts of the N parts of data to the hosts corresponding to other devices in the data parallel group, where N is the number of devices in the data parallel group.

[0133] For example, in step S301, the "corresponding host" refers to a host that is physically connected to the device, such as a host connected through a switch. For example, in the data parallel group related to the current stage, the following operations are performed: each device in the data parallel group first completes the forward calculation and reverse calculation locally to obtain gradient data. When the reverse calculation task is completed, each device divides all the gradient data it generates into N copies of data, and transmits one of the copies of data to the host physically connected to it through D2H transmission, and sends the remaining N-1 copies of data to the hosts corresponding to other devices in the data parallel group. For the convenience of description, the device that sends data is referred to as the source device, and the host that receives the data is referred to as the destination host. If the source device and the destination host are not in the same computing node, the source device can send the gradient data through network communication. If they are in the same computing node, the source device can send the gradient data through the intra-machine interconnection. For example, in the same computing node, multiple devices can be connected through a high-speed bus. An example of a source device sending gradient data through the intra-machine interconnection in the same computing node is as follows: Figure 2B Taking device 0 as the source device and host 1 as the destination host as an example, device 0 can send gradient data to host 0 via D2H transmission, and then send the gradient data to host 1 via the UPI bus (for example, the transmission between host 0 and host 1 can be implicit and does not require human control); alternatively, device 0 can send gradient data to device 4 via the high-speed bus, and device 4 can then send the gradient data to host 1 via D2H transmission.

[0134] Corresponding to step S301, an example of the above step S1032 may include performing the following operations in each data parallel group related to the current stage: the host corresponding to each device in the data parallel group executes the following step S302.

[0135] Step S302: The host performs an optimization update task based on all received gradient data, obtains at least some updated parameters, and sends at least some updated parameters to all devices in the data parallel group for use in the next round of iterative training.

[0136] For example, in step S302, for each host (referring to the host corresponding to each device in the data parallel group), all the received gradient data include a copy of the data transmitted by the device physically connected to it, and the data sent by other devices in the data parallel group (a total of N-1 copies). Each host can perform optimization and update tasks based on all the gradient data it has received, obtain some updated parameters, and send them to all devices in the data parallel group. For the convenience of description, the host that sends the parameters is referred to as the source host, and the device that receives the parameters is referred to as the destination device. If the source host and the destination device are not in the same computing node, the source host can send the updated parameters through network communication. If they are in the same computing node, the source host can send the updated parameters through the intra-machine interconnection. For example, in the same computing node, multiple devices can be connected through a high-speed bus. An example of the source host sending updated parameters through the intra-machine interconnection in the same computing node is as follows: Figure 2B For example, taking host 0 as the source host and device 4 as the destination device, host 0 can send the updated parameters to device 0 via H2D transmission, and device 0 can send the updated parameters to device 4 via the high-speed bus. Alternatively, host 0 can send the updated parameters to host 1 via the UPI bus, and host 1 can then send the updated parameters to device 4 via H2D transmission. In this way, each device in the data parallel group can obtain the complete updated parameters for use in the next round of iterative training.

[0137] By directly dividing data at the device and distributing it to different hosts in the above manner, the amount of data transmitted to the corresponding host can be reduced, the time overhead of the data transmission process can be effectively reduced, and the data transmission efficiency can be further improved.

[0138] Figure 5 An exemplary schematic diagram of a distributed training method provided for at least one embodiment of the present disclosure. Figure 5 This is an example of the above steps S301 to S302.

[0139] For example, Figure 5 As shown in FIG, a data parallel group includes four devices (device 0 to device 3). Here, it is assumed that the four devices are located on different computing nodes.

[0140] For example, Figure 5 As shown, in response to the completion of the reverse calculation task, device 0 divides all the gradient data it generates (stored in the device memory) into 4 parts of data ( Figure 5 Data a0 to data a3 in the D2H transmission, and transfer one of the data a0 to the corresponding host (that is, Figure 5Device 0 is connected to device 0 via a switch and stored in the host memory. Device 0 sends data a1 through a3 to the hosts corresponding to devices 1 through 3 via network communication.

[0141] In order to avoid the information in the figure being too complicated and affecting understanding, Figure 5 Only the process of device 0 sending gradient data is shown, while the process of devices 1 to 3 sending gradient data is not shown. These processes will be described in detail below.

[0142] In response to the completion of the reverse calculation task, device 1 divides all the gradient data it generates (stored in the device memory) into 4 parts of data ( Figure 5 Data b0 to data b3 in the D2H transmission, and transfer one of the data b1 to the corresponding host (that is, Figure 5 Device 1 is connected to host 1 via a switch and stores the data in the host memory. Device 1 sends data b0, b2, and b3 to the hosts corresponding to device 0, device 2, and device 3, respectively, through network communication.

[0143] In response to the completion of the reverse calculation task, device 2 divides all the gradient data it generates (stored in the device memory) into 4 parts of data ( Figure 5 Data c0 to data c3 in the D2H transmission, and transfer one of the data c2 to the corresponding host (that is, Figure 5 Device 2 is connected to host 2 via a switch and stores the data in the host memory. Device 2 sends data c0, data c1, and data c3 to the hosts corresponding to device 0, device 1, and device 3, respectively, through network communication.

[0144] In response to the completion of the reverse calculation task, device 3 divides all the gradient data it generates (stored in the device memory) into 4 parts of data ( Figure 5 Data d0 to data d3 in the D2H transmission, and transfer one of the data d3 to the corresponding host (that is, Figure 5 Device 3 is connected to host 3 via a switch and stores the data in the host memory. Device 3 sends data d0 through d2 to the hosts corresponding to devices 0 through 2, respectively, via network communication.

[0145] Through the above operations, host 0's host memory stores data a0, b0, c0, and d0. Host 0 performs an optimization update task based on this data, obtaining updated parameter p0. This parameter is then transferred to device 0 via H2D transmission and then broadcast to devices 1 through 3 via network communication. Host 1's host memory stores data a1, b1, c1, and d1. Host 1 performs an optimization update task based on this data, obtaining updated parameter p1. This parameter is then transferred to device 1 via H2D transmission and then broadcast to devices 0, 2, and 3 via network communication. The same applies to hosts 2 and 3 and is not further described here. Through the above operations, each device obtains updated parameters p0 through p3 for use in the next round of iterative training.

[0146] It should be noted that the above method is not only applicable to the architecture adopting the data parallel-pipeline parallel strategy, but can also be further applied to the architecture adopting the tensor parallel-data parallel-pipeline parallel strategy.

[0147] In the distributed training method provided in at least one embodiment of the present disclosure, network communication can be based on the remote direct memory access (RDMA) mechanism. For example, the Ethernet-based RDMA (RDMA over Converged Ethernet v2, RoCEv2) network protocol can support the RDMA mechanism. For example, the broadcast transmission described above can be implemented through the RDMA mechanism. By utilizing the broadcast function of RDMA, the number of moves required during data transmission can be significantly reduced. In large-scale distributed training scenarios (for example, N>16), the RDMA broadcast mechanism can reduce the number of data moves to the original 1 / (N-1), thereby greatly improving data transmission efficiency. In theory, the larger the scale, the lower the communication overhead.

[0148] In the distributed training method provided in at least one embodiment of the present disclosure, a return pipeline (U-shaped pipeline) may also be constructed. For example, the distributed training method may further include the following step S104.

[0149] Step S104: assign the i-th stage and the M-i+1-th stage to the same computing node, where 0<i≤M / 2, M is the total number of stages into which the target model is divided, and the i-th stage represents the i-th stage executed sequentially during the forward propagation of the target model.

[0150] For example, in step S104, each computing node can undertake two stages of computing tasks, and the computing tasks of the two stages are staggered in time. In this way, the first stage (that is, the first stage above) and the last stage (in the case of M stages, the last stage is the Mth stage) of the target model can be assigned to the same computing node. For the convenience of description, it is assumed that the devices in a computing node are divided into two groups, where the first group of devices undertakes the computing tasks of the first stage of the target model, and the second group of devices undertakes the computing tasks of the last stage. In the forward computing stage, when the second group of devices starts to execute the forward computing tasks, the forward computing tasks of the first group of devices have been completed and stopped running. At this time, the first group of devices will not compete with the second group of devices for computing resources and communication resources, so that the computing resources and communication resources (such as available network bandwidth) available to the second group of devices are increased to twice the original amount. The same principle applies to the reverse computing phase. When the first group of devices begins executing the reverse computing task, the second group of devices has already completed its reverse computing task and ceased execution. At this point, the second group of devices no longer competes with the first group for computing and communication resources, doubling the computing and communication resources (e.g., available network bandwidth) available to the first group of devices. For example, step S104 can be applied to architectures employing a tensor parallelism-pipeline parallelism strategy, or to architectures employing a tensor parallelism-data parallelism-pipeline parallelism strategy.

[0151] Through the above method, the available computing resources and communication resources of the computing nodes in the first stage can be effectively improved, further alleviating the communication bottleneck problem in large-scale distributed deep learning training.

[0152] Figure 6 A schematic diagram of a return pipeline provided in accordance with at least one embodiment of the present disclosure.

[0153] For example, Figure 6 As shown in the figure, this example uses TP4-PP8 (tensor parallelism is 4, pipeline parallelism is 8) for partitioning. Figure 6 This is a simplified diagram that only shows the devices in each computing node. Components such as the host and network card are not shown in the diagram. Figure 6 Four computing nodes (computing node 0 to computing node 3) are shown, and each computing node includes 8 devices (device 0 to device 7). Figure 6 Each dotted ellipse box represents a stage of the target model, and the order of the arrows represents the execution order of each stage during forward calculation. Figure 6The figure shows eight phases, each of which uses four devices to implement tensor parallelism. For example, devices 0 through 3 on compute node 0 implement tensor parallelism in the first phase, devices 0 through 3 on compute node 1 implement tensor parallelism in the second phase, and so on, devices 4 through 7 on compute node 0 implement tensor parallelism in the eighth phase. This shows that both the first phase (the first phase) and the last phase (the eighth phase) of the target model are executed on compute node 0. During the forward computation phase, when devices 4 through 7 on compute node 0 begin executing forward computation tasks, the forward computation tasks on devices 0 through 3 on compute node 0 have already completed and ceased. At this point, devices 0 through 3 no longer compete with devices 4 through 7 for computing and communication resources, doubling the computing and communication resources available to devices 4 through 7. The same principle applies during the reverse computation phase. When devices 0 through 3 on compute node 0 begin executing the reverse computation task, devices 4 through 7 on compute node 0 have already completed their reverse computation tasks and ceased execution. At this point, devices 4 through 7 no longer compete with devices 0 through 3 for computing and communication resources, doubling the computing and communication resources available to devices 0 through 3. For example, assuming each compute node has eight network cards, during the reverse computation phase, devices 0 through 3 can use all eight cards for communication, doubling the available network bandwidth and effectively improving network bandwidth utilization.

[0154] It should also be noted that, in various embodiments of the present disclosure, the order in which the various steps of the distributed training method are executed is not limited. Although the execution process of the various steps is described above in a specific order, this does not constitute a limitation on the embodiments of the present disclosure. The various steps in the distributed training method can be executed serially or in parallel, which can be determined according to actual needs.

[0155] For example, compared with the above description, the distributed training method provided in at least one embodiment of the present disclosure may also include more or fewer steps, and the embodiments of the present disclosure are not limited to this.

[0156] At least one embodiment of the present disclosure also provides a distributed training system. Figure 2A and Figure 2B At least one embodiment of the present disclosure provides a distributed training system comprising a plurality of computing nodes, each of which comprises at least one host and at least one device. The distributed training system is configured to perform distributed training on a target model.

[0157] For example, the target model is divided into multiple stages in a pipelined parallel mode, each stage including at least one layer of the target model, and the multiple stages are distributed across multiple devices in multiple computing nodes. In the current round of iterative training, the training of at least one stage of the target model is collaboratively performed by at least one corresponding device and at least one corresponding host.

[0158] For each stage of the target model, the device is configured to: perform a forward calculation task to obtain a forward calculation result; perform a reverse calculation task based on the forward calculation result to obtain gradient data; in response to the completion of the reverse calculation task, trigger the execution of the optimization update task and obtain the updated parameters for use in the next round of iterative training.

[0159] For at least one stage of the target model, the device is further configured to: in response to the completion of the reverse calculation task, transfer at least part of the gradient data to the host and trigger the host to perform the optimization update task; the host is configured to: perform the optimization update task based on at least part of the gradient data, obtain at least part of the updated parameters, and transfer at least part of the updated parameters to the device for use in the next round of iterative training.

[0160] For example, in at least one embodiment of the present disclosure, the devices in the multiple computing nodes are divided into multiple data parallel groups in a data parallel mode, and gradient aggregation is performed between the devices in the same data parallel group. For each data parallel group associated with at least one of the above-mentioned stages, each device in the data parallel group is further configured to: in response to the completion of the reverse calculation task, transmit the gradient data to the corresponding host through D2H transmission. In each data parallel group associated with at least one of the above-mentioned stages, the host corresponding to each device in the data parallel group is further configured to: divide the gradient data transmitted by the device into N parts of data, and send N-1 parts of the N parts of data to the hosts corresponding to other devices in the data parallel group, respectively, where N is the number of devices in the data parallel group; perform optimization update tasks based on all the received gradient data to obtain at least part of the updated parameters; obtain the updated parameters generated by the hosts corresponding to other devices in the data parallel group, and transmit all the updated parameters to the device through H2D transmission for use in the next round of iterative training.

[0161] For example, in at least one embodiment of the present disclosure, for each data parallel group associated with at least one of the aforementioned stages, each device in the data parallel group is further configured to: in response to the completion of the reverse calculation task, divide the gradient data into N parts of data, transmit one of the N parts of data to the corresponding host via D2H transmission, and send the remaining N-1 parts of the N parts of data to the hosts corresponding to other devices in the data parallel group, where N is the number of devices in the data parallel group. In each data parallel group associated with at least one of the aforementioned stages, the host corresponding to each device in the data parallel group is further configured to: perform an optimization update task based on all received gradient data, obtain at least some updated parameters, and send at least some of the updated parameters to all devices in the data parallel group for use in the next round of iterative training.

[0162] The distributed training system provided by at least one embodiment of the present disclosure also includes an allocation module, which is configured to allocate the i-th stage and the M-i+1-th stage to the same computing node, where 0<i≤M / 2, M is the total number of stages into which the target model is divided, and the i-th stage represents the i-th stage executed sequentially during the forward propagation process of the target model.

[0163] For example, in at least one embodiment of the present disclosure, at least one stage includes a first stage, and the first stage includes the frontmost layer in the target model in the forward propagation order.

[0164] For example, in at least one embodiment of the present disclosure, the configuration of the host corresponding to the first stage is higher than the configuration of the host corresponding to other stages.

[0165] For example, in at least one embodiment of the present disclosure, each computing node further includes at least one network card, and at least one host is configured to drive the at least one network card to implement network communication among the multiple computing nodes.

[0166] For example, in at least one embodiment of the present disclosure, network communication is based on a remote direct memory access mechanism.

[0167] It should be noted that the above-mentioned various modules can be implemented by software, hardware, firmware or any combination thereof. For example, the distribution module can be implemented as a distribution circuit, and the embodiments of the present disclosure do not limit the specific implementation methods thereof.

[0168] It should be understood that the distributed training system provided in at least one embodiment of the present disclosure can be used to implement the aforementioned distributed training method, and can also achieve technical effects similar to those of the aforementioned distributed training method, which will not be elaborated here.

[0169] It should be noted that in the embodiments of the present disclosure, the distributed training system may include more or fewer modules or units, and the connection relationship between the modules or units is not limited and can be determined according to actual needs. The specific configuration of each module or unit is not limited and can be composed of analog devices based on circuit principles, or digital chips, or other applicable methods.

[0170] Figure 7 A schematic block diagram of an electronic device provided in accordance with at least one embodiment of the present disclosure.

[0171] For example, Figure 7 As shown, the electronic device 700 includes at least one processor 701 and at least one memory 702. The at least one memory 702 includes one or more computer program modules. The one or more computer program modules are stored in the memory 702 and configured to be executed by the at least one processor 701. The one or more computer program modules include instructions for executing the above-mentioned distributed training method. When executed by the at least one processor 701, one or more steps in the distributed training method provided in at least one embodiment of the present disclosure can be performed. The memory 702 and the processor 701 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0172] For example, the processor 701 may be a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), an artificial intelligence (AI) accelerator, or other processing units with data processing and / or program execution capabilities, such as a field programmable gate array (FPGA). For example, the central processing unit (CPU) may be an X86, ARM, or RISC-V architecture. The processor 701 may be a general-purpose processor or a dedicated processor, and may control other components in the electronic device 700 to perform desired functions.

[0173] For example, the memory 702 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, a flash memory, and the like.

[0174] Figure 8A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0175] The electronic devices in at least one embodiment of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0176] An electronic device includes at least one processor and memory. The processor herein may be referred to as processing device 801, described below. The memory may include at least one of read-only memory (ROM), random access memory (RAM), and storage device 808, described below. The memory is used to store programs for executing the methods described in the various method embodiments described above; the processor is configured to execute the programs stored in the memory. The processor may include a central processing unit (CPU) or other processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0177] like Figure 8 As shown, electronic device 800 may include a processing device 801 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) or programs loaded from a storage device 808 into a random access memory (RAM). Various programs and data required for the operation of electronic device 800 are also stored in RAM 803. Processing device 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface is also connected to bus 804.

[0178] Typically, the following devices may be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a display, a speaker, a vibrator, etc.; a storage device 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Figure 8 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0179] In particular, according to at least one embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, at least one embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of at least one embodiment of the present disclosure are performed.

[0180] It should be noted that the computer-readable medium described above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In at least one embodiment of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. Furthermore, in at least one embodiment of the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, radio frequency (RF), or any suitable combination thereof.

[0181] The computer-readable medium may be included in the electronic device 800 , or may exist independently without being incorporated into the electronic device 800 .

[0182] Figure 9 A schematic block diagram of a non-transitory computer-readable storage medium is provided for at least one embodiment of the present disclosure.

[0183] For example, Figure 9 As shown, a non-transitory computer-readable storage medium 900 stores computer-readable instructions 901 , which, when executed by at least one processor, perform one or more steps of the above-mentioned distributed training method.

[0184] For example, the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a flash memory, or any combination of the above storage media, or other suitable storage media. For example, the readable storage medium may also be Figure 7 For the memory 702 in the embodiment, the related description can be referred to the aforementioned content and will not be repeated here.

[0185] Although the present disclosure has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications or improvements may be made based on the embodiments of the present disclosure. Therefore, such modifications or improvements, as long as they do not depart from the spirit of the present disclosure, are within the scope of protection claimed by the present disclosure.

[0186] Regarding this disclosure, the following points need to be explained:

[0187] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure. Other structures may refer to conventional designs.

[0188] (2) For the sake of clarity, the thickness of layers or regions in the drawings used to describe the embodiments of the present disclosure are enlarged or reduced, that is, these drawings are not drawn according to the actual scale.

[0189] (3) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to form new embodiments.

[0190] The above description is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A distributed training method, applied to a distributed training system, wherein: The distributed training system is configured to perform distributed training on a target model. The distributed training system includes multiple computing nodes, each computing node includes at least one host and at least one device, each computing node also includes at least one network card, and the at least one host is configured to drive at least one network card to achieve network communication between the multiple computing nodes. The target model is divided into multiple stages in a pipeline parallel mode, each stage includes at least one layer of the target model, and the multiple stages are respectively distributed in multiple devices of the multiple computing nodes. In the current round of iterative training, at least one stage of training of the target model is collaboratively performed by at least one corresponding device and at least one corresponding host. The distributed training method includes: For each stage of the target model, The device performs a forward calculation task to obtain a forward calculation result; The device performs a reverse calculation task based on the forward calculation result to obtain gradient data; In response to the completion of the reverse calculation task, the device immediately triggers the execution of the optimization update task and obtains the updated parameters for use in the next round of iterative training; For the first stage of the target model, in response to the completion of the reverse calculation task, the device immediately triggers execution of an optimization update task and obtains updated parameters for use in the next round of iterative training, including: In response to the completion of the reverse calculation task, the device immediately transmits at least part of the gradient data to the host and triggers the host to perform the optimization update task; The host performs an optimization update task based on at least part of the gradient data to obtain at least part of the updated parameters, and transmits the at least part of the updated parameters to the device for use in the next round of iterative training. The first stage is the first stage executed sequentially during the forward propagation process of the target model.

2. The method according to claim 1, wherein The devices in the multiple computing nodes are divided into multiple data parallel groups in a data parallel mode, and gradient aggregation is performed between devices in the same data parallel group. In response to the completion of the reverse calculation task, the device immediately transmits at least part of the gradient data to the host, including: For each data parallel group associated with the first stage, In response to the completion of the reverse calculation task, each device in the data parallel group immediately transmits the gradient data to the corresponding host through device-to-host transmission.

3. The method according to claim 2, wherein: The host performs an optimization update task based on at least part of the gradient data to obtain at least part of updated parameters, and transmits the at least part of the updated parameters to the device for use in a next round of iterative training, including: In each data parallel group associated with the first stage, for each host corresponding to each device in the data parallel group, The host divides the gradient data transmitted by the device into N parts of data, and sends N-1 parts of the N parts of data to the hosts corresponding to other devices in the data parallel group, where N is the number of devices in the data parallel group; The host performs an optimization and updating task based on all received gradient data to obtain at least some updated parameters; The host obtains updated parameters generated by the hosts corresponding to other devices in the data parallel group, and transmits all updated parameters to the devices through host-to-device transmission for use in the next round of iterative training.

4. The method according to claim 1, wherein The devices in the multiple computing nodes are divided into multiple data parallel groups in a data parallel mode, and gradient aggregation is performed between devices in the same data parallel group. In response to the completion of the reverse calculation task, the device immediately transmits at least part of the gradient data to the host, including: For each data parallel group associated with the first stage, In response to the completion of the reverse calculation task, each device in the data parallel group immediately divides the gradient data into N parts of data, transfers one of the N parts of data to the corresponding host through device-to-host transmission, and sends the remaining N-1 parts of the N parts of data to the hosts corresponding to other devices in the data parallel group, where N is the number of devices in the data parallel group.

5. The method according to claim 4, wherein The host performs an optimization update task based on at least part of the gradient data to obtain at least part of updated parameters, and transmits the at least part of the updated parameters to the device for use in a next round of iterative training, including: In each data parallel group associated with the first stage, for each host corresponding to each device in the data parallel group, The host performs an optimization update task based on all received gradient data, obtains at least part of the updated parameters, and sends the at least part of the updated parameters to all devices in the data parallel group for use in the next round of iterative training.

6. The method according to claim 1, further comprising: Assign the i-th stage and the M-i+1-th stage to the same computing node, Wherein, 0<i≤M / 2, M is the total number of stages into which the target model is divided, and the i-th stage represents the i-th stage executed sequentially during the forward propagation process of the target model.

7. The method according to claim 1, wherein The configuration of the host corresponding to the first stage is higher than the configuration of the hosts corresponding to other stages.

8. The method according to claim 1, wherein The network communication is based on a remote direct memory access mechanism.

9. A distributed training system configured to perform distributed training on a target model, wherein: The distributed training system includes: a plurality of computing nodes, each computing node comprising at least one host and at least one device, each computing node further comprising at least one network card, the at least one host being configured to drive the at least one network card to implement network communication among the plurality of computing nodes, wherein the target model is divided into a plurality of stages in a pipeline parallel mode, each stage comprising at least one layer of the target model, and the plurality of stages are respectively distributed in a plurality of devices of the plurality of computing nodes; In the current round of iterative training, at least one stage of training of the target model is collaboratively performed by at least one corresponding device and at least one corresponding host. For each stage of the target model, the device is configured to: Execute the forward calculation task and obtain the forward calculation result; Performing a reverse calculation task based on the forward calculation result to obtain gradient data; In response to the completion of the reverse calculation task, immediately triggering the execution of the optimization update task and obtaining the updated parameters for use in the next round of iterative training; For the first stage of the target model, the device is further configured to: in response to the completion of the reverse calculation task, immediately transmit at least part of the gradient data to the host, and trigger the host to perform the optimization update task; the host is configured to: perform the optimization update task based on at least part of the gradient data, obtain at least part of the updated parameters, and transmit the at least part of the updated parameters to the device for use in the next round of iterative training, The first stage is the first stage executed sequentially during the forward propagation process of the target model.

10. An electronic device comprising: at least one processor; at least one memory including one or more computer program modules; The one or more computer program modules are stored in the at least one memory and configured to be executed by the at least one processor, and the one or more computer program modules are used to implement the distributed training method according to any one of claims 1 to 9.

11. A non-transitory computer-readable storage medium having computer instructions stored thereon, wherein: When the computer instructions are executed by at least one processor, the distributed training method according to any one of claims 1 to 9 is performed.

Images