Method and system for parallel training of dnn model facing multi-machine multi-card computing system

Abstract

The application provides a pipeline parallel method and system for a multi-machine multi-card system, wherein each Mini-batch is divided into T Micro-batches and sequentially transmitted into a pipeline structure composed of N GPUs on each computing node, and Micro-batches from different Mini-batches are alternately executed in the pipeline in a forward propagation and a backward propagation mode. On any computing node, the pipeline structure is completely the same and almost no "bubble" is generated, so that a very high GPU utilization rate can be generated. In order to overcome the weight inconsistency and weight obsolescence problems caused by the asynchronous update of Stage weights, for any GPU of each computing node, the first Micro-batch in each Mini-batch is selected as a "leader" to be responsible for weight prediction, so that all Micro-batches in each Mini-batch use nearly consistent Stage weights when performing forward propagation and backward propagation, and the effectiveness of model parameter learning in the pipeline parallel training process is effectively guaranteed.

Description

Technical Field

[0001] This invention belongs to the field of deep learning, and in particular relates to a method and system for parallel training of DNN models for multi-machine, multi-GPU computing systems. Background Technology

[0002] As a crucial supporting technology in the field of artificial intelligence, deep learning has been widely applied in various areas such as image recognition, machine translation, and speech recognition. With the increasing complexity of problem-solving, the depth of neural networks is constantly increasing, and the scale of DNN model parameters is also growing rapidly. However, in stark contrast to the ever-increasing scale of neural networks, the growth rate of GPU storage capacity is very slow. Due to storage capacity limitations, it is no longer possible to train large-scale deep neural network models using a single GPU. For example, training a large model like GPT-3, which contains 175 billion parameters, requires 350GB and 175GB of storage space when storing data with 32-bit and 16-bit precision, respectively. Common GPUs only have storage capacities of tens of gigabytes; for example, the Nvidia GeForce RTX 3080 Ti has 12GB of VRAM, and the Nvidia GeForce RTX 3090 has 32GB. Clearly, it is no longer possible to train such large-scale deep neural networks on a single GPU.

[0003] Pipeline parallelism overcomes the limitations of single GPU storage space by dividing a DNN model into multiple stages and training them collaboratively on different GPUs. Because pipeline parallelism has low communication overhead and overcomes the limitation of single GPU memory capacity, it has become an effective method for training large-scale deep neural networks. A major drawback of pipelined parallel methods is low GPU utilization. This is because, during the forward and backward propagation of a mini-batch, only one GPU is active in any given pipeline time unit, while the others remain idle. To accelerate pipelined parallel training, many efficient pipelined parallel methods have been proposed, among which the two most well-known and representative are GPipe (proposed by Google) and PipeDream (proposed by Microsoft in collaboration with Stanford). GPipe divides a mini-batch into several smaller micro-batches and allows these micro-batches to be executed in parallel within the pipeline to improve parallelism. GPipe is a synchronous pipelined parallel method, whose advantage lies in the synchronous updating of model parameters, which does not affect the convergence characteristics of model training. However, due to the large number of "bubbles" in GPipe's pipeline structure, its GPU utilization remains low, which correspondingly limits further improvements in GPipe's throughput. PipeDream is an asynchronous pipelined parallel method. It allows all mini-batches to be trained in a "1F1B" (One Forward One Backward) manner within the pipeline. When the pipeline reaches steady state, all GPUs in the pipeline are performing forward or backward propagation at any given pipeline time unit. This training method results in virtually no "bubbles" in the pipeline, leading to extremely high GPU utilization and consequently, very high throughput. Although PipeDream uses a weight stashing strategy to ensure consistent network weights for forward and backward propagation in each mini-batch, it does not effectively address the weight staleness problem caused by asynchronous model updates. This negatively impacts model convergence and accuracy.

[0004] In summary, pipelined parallelism is an effective method for training large-scale DNN models. However, existing pipelined parallelism methods such as GPipe and PipeDream fail to achieve a good trade-off between GPU utilization and model convergence properties. This necessitates an efficient model parallelism method that can fully leverage the parallel computing capabilities of multi-GPU computing platforms to efficiently train large-scale DNN models. Summary of the Invention

[0005] The technical problem to be solved by this invention is how to train large-scale deep neural network models with high GPU utilization, good model convergence, and high accuracy. A pipelined parallel method and system for multi-machine and multi-GPU computing systems is proposed.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A method for pipelined parallel training of DNN models for multi-machine, multi-GPU computing systems includes the following steps:

[0008] Step 1: Given a multi-machine, multi-GPU computing system containing M×N GPUs, where M is the number of computing nodes and N is the number of GPUs on each computing node;

[0009] Step 2: Vertically divide the DNN model into N Stages corresponding to the number of GPUs on each computing node, load them onto the GPUs corresponding to each computing node, and then initialize the learning rate and optimizer parameters of the DNN model.

[0010] Step 3: Obtain a global batch (Global Mini-batch, GMB) and divide it into M parts, then distribute them to each computing node. Each computing node will receive one mini-batch of batch data.

[0011] Step 4: On each computing node, each batch of data Mini-batch is divided into T Micro-batch, which are then launched sequentially into a pipeline structure composed of N GPUs. Each Micro-batch is arranged in the N GPUs in an alternating manner of forward propagation and backward propagation.

[0012] Step 5: Training ends after all global batch data has been trained on the multi-machine, multi-card system.

[0013] Furthermore, on each computing node, the T micro-batches divided from each mini-batch use consistent prediction weights during forward and backward propagation.

[0014] Furthermore, the consistent weights are predicted by the first Micro-batch in each Mini-batch segment as the “leader”, and the other Micro-batches segmented from that Mini-batch use the weights predicted by the “leader” during forward and backward propagation.

[0015] Furthermore, the weights for the first micro-batch prediction are calculated as follows:

[0016]

[0017] in, The weights generated by prediction, W tΔW represents the current weight corresponding to the t-th pipeline time unit, where lr is the learning rate; s refers to the number of Stage weight updates between the current pipeline time unit and the pipeline unit corresponding to the Mini-batch (the "leader") after backpropagation ends. t This represents the relative change in the current weight after one weight update.

[0018] Furthermore, for the i-th GPU on any computing node, after all micro-batches in a mini-batch have completed backpropagation, the i-th GPU on all computing nodes uses the accumulated gradient values ​​and performs global gradient summation through AllReduce, a communication operation method for distributed deep learning, and then performs weight updates on the corresponding stage.

[0019] Furthermore, in pipelined parallel training, each computing node is assigned N processes. Each process j corresponds one-to-one with the GPU on its computing node through the jmod M operation. The process is used to perform the forward and backward propagation calculations of the Stage and to implement global gradient summation within the communication group. The communication group refers to a group composed of processes with the same j mod M value on all computing nodes.

[0020] Furthermore, the process generates several auxiliary threads, which are used to transfer activation values ​​and gradient values ​​between GPUs, and each auxiliary thread is responsible for sending or receiving a tensor data.

[0021] Furthermore, on any computing node, each process creates and manages a first-in-first-out queue (actQ) to store the activation values ​​input by the Stage.

[0022] This invention also provides a pipelined parallel training system for multi-GPU computing platforms, characterized in that, given a multi-machine, multi-GPU computing system containing M×N GPUs, where M is the number of computing nodes and N is the number of GPUs on each computing node, it includes the following modules:

[0023] Model splitting module: This module is used to vertically split the DNN model into N stages corresponding to the number of GPUs on each computing node, load them onto the GPUs corresponding to each computing node, and then initialize the learning rate and optimizer parameters for training the DNN model.

[0024] Input module: used to acquire global batch processing data and divide the global batch processing data into M parts and distribute them to each computing node, with each computing node obtaining a batch of data Mini-batch;

[0025] Parallel processing module: On each computing node, each batch of data Mini-batch is divided into T Micro-batch, which are then launched sequentially into a pipeline structure composed of N GPUs. Each Micro-batch is arranged in the N GPUs in an alternating manner of forward propagation and backward propagation.

[0026] Training termination module: Training ends when all global batch data has been trained on the multi-machine, multi-card system.

[0027] By adopting the above technical solution, the present invention has the following beneficial effects:

[0028] This invention provides a pipelined parallelism method and system for multi-machine, multi-GPU systems. First, a global mini-batch is selected and divided according to the number of computing nodes M. Each computing node is responsible for processing one mini-batch. Then, on each computing node, each mini-batch is divided into T micro-batches and sequentially launched into a pipeline structure composed of N GPUs. This allows micro-batches from different mini-batches to be executed alternately in the pipeline using a forward and backward propagation method. On any given computing node, the pipeline structure is identical and produces almost no "bubbles," thus achieving extremely high GPU utilization. Furthermore, the pipeline structure proposed in this invention does not change with the number of micro-batches into which each mini-batch is divided. This means that when the number of stages into which a DNN model is divided is fixed, the pipeline can always produce consistently high GPU utilization. To overcome the problems of weight inconsistency and stale weights caused by asynchronous Stage weight updates, this application selects the first Micro-batch in each Mini-batch as the "leader" to be responsible for weight prediction for any GPU on each computing node. This ensures that all Micro-batch in each Mini-batch use nearly consistent Stage weights during forward and backward propagation, effectively guaranteeing the effectiveness of model parameter learning during pipelined parallel training. Attached Figure Description

[0029] Figure 1 A system flowchart provided for embodiments of the present invention;

[0030] Figure 2 A schematic diagram illustrating the loading of the corresponding GPUs into Stage on a multi-machine, multi-GPU computing system consisting of M×4 GPUs.

[0031] Figure 3(a), (b), and (c) respectively illustrate the pipeline structure diagrams on each computing node when T=4, T=2, and T=1 in a multi-machine, multi-card computing system;

[0032] Figure 4 This is a schematic diagram of the communication group consisting of all processes on a multi-machine, multi-card system composed of M×4 GPUs;

[0033] Figure 5 A schematic diagram of tensor transfers between GPUs on each compute node. Detailed Implementation

[0034] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Figures 1 to 5 This invention illustrates a specific implementation of a pipelined parallel method for multi-machine, multi-GPU computing systems, comprising the following steps:

[0036] Step 1: Given a multi-machine, multi-GPU computing system containing M×N GPUs, where M is the number of computing nodes and N is the number of GPUs on each computing node.

[0037] Step 2: Vertically divide the DNN model into N Stages corresponding to the number of GPUs on each computing node, load them onto the GPUs corresponding to each computing node, and then initialize the learning rate and optimizer parameters of the DNN model.

[0038] In this embodiment, before the pipelined parallel training begins, PipeDream's model partitioning method is used to partition a DNN model into N Stages. These N Stages are then loaded onto N different GPUs on all computing nodes. In the system implementation, the number of Stages a DNN model is partitioned into is equal to the total number of GPUs and processes on each computing node, and they correspond one-to-one. The system implementation is based on the deep learning framework PyTorch. During pipelined parallel training, each computing node is assigned N processes. Each process j corresponds one-to-one with a GPU on its computing node through a j mod M operation. These processes are used to perform forward and backward propagation calculations for the Stages and to perform global gradient summation within a communication group. The communication group refers to a group consisting of processes on all computing nodes with the same j mod M value. Figure 4As shown, processes 0, 4, 8, ..., M×N-4 form communication group 0; processes 1, 5, 9, ..., M×N-3 form communication group 1; processes 2, 6, 10, ..., M×N-2 form communication group 2; and processes 3, 7, 11, ..., M×N-1 form process group 3. Each process generates several auxiliary threads, which are responsible for transferring activation and gradient values ​​between GPUs. Each auxiliary thread is only responsible for sending or receiving tensor data of a single type. By partitioning the model and performing pipelined parallel computation, the goal of accelerating the training of large-scale DNN models is achieved. Figure 2 This document describes the loading of a DNN model into four stages on a multi-machine, multi-GPU computing system consisting of M×4 GPUs. In the pipelined parallel training, all processes have two total variables: `fwd_idx` representing the number of micro-batches during forward propagation and `bwd_idx` representing the number of micro-batches during backpropagation. Furthermore, each process needs to create and manage a First-In-First-Out (FIFO) queue `actQ` to store the activation values ​​input to the stage.

[0039] This embodiment uses recomputing to reduce the storage overhead of activation values. In the Error Back Propagation (BP) algorithm, the i-th neural network layer needs the gradient value of the (i+1)-th layer and the activation value of the i-th layer during backpropagation. To ensure that each Stage uses the exact same activation value intermediate variables during backpropagation as during forward propagation, each GPU needs to store the intermediate activation value variables of all neural network layers contained in the corresponding Stage, which obviously consumes a lot of GPU storage space. Specifically, each GPU does not store the activation values ​​of all layers contained in the Stage, but only the activation values ​​of the Stage's input layer. When a Stage is ready to backpropagate, the input layer activation values ​​and prediction weights are used to perform a forward propagation again, thereby restoring the intermediate variable activation values ​​of all layers contained in this Stage. Before the pipelined parallel training begins, each process needs to start an auxiliary thread equal to the number of communication tensors, and each auxiliary thread is responsible for sending or receiving a specific tensor. The auxiliary thread includes four types of tensor communication operations: sending activation values ​​during forward propagation, receiving activation values ​​during forward propagation, sending activation values ​​during backward propagation, and receiving gradient values ​​during backward propagation. PyTorch's distributed package (torch.distributed) supports three built-in backends—GLOO, MPI, and NCCL—to implement distributed tensor communication. This invention uses PyTorch's GLOO backend to implement inter-process communication operations. It's important to note that since the GLOO backend does not support direct GPU-to-GPU communication, but only CPU-to-CPU communication, tensor communication between GPUs requires using the CPU as a "stepping stone." Figure 5 As shown, for any GPU, when a tensor on this GPU needs to be sent to another GPU, the auxiliary thread first needs to copy the tensor from the GPU to the CPU, and then the auxiliary thread on the target GPU loads the tensor from the CPU to the target GPU.

[0040] Step 3: Obtain a global batch (Global Mini-batch, GMB) and divide it into M parts, then distribute them to each computing node. Each computing node obtains one mini-batch of batch data.

[0041] Step 4: On each computing node, each batch of data Mini-batch is divided into T Micro-batch, which are then launched sequentially into a pipeline structure composed of N GPUs. Each Micro-batch is arranged in the N GPUs in an alternating manner of forward propagation and backward propagation.

[0042] In this embodiment, on each computing node, the T Micro-batches divided from each Mini-batch use consistent prediction weights during forward and backward propagation.

[0043] At any computing node, the pipeline structure in this embodiment sequentially fires the T smaller micro-batches into the pipeline structure. The micro-batches from multiple mini-batches are then executed alternately in a forward propagation and a backward propagation (i.e., micro-batch "1F1B") manner, resulting in identical execution with almost no "bubbles" and thus extremely high GPU utilization. The pipeline structure proposed in this invention does not change with the number of micro-batches into which each mini-batch is divided. This means that when the number of stages into which a DNN model is divided is fixed, the pipeline can always produce consistently high GPU utilization, unaffected by T.

[0044] In this embodiment, the unified weights are predicted by the first Micro-batch in each Mini-batch segment, which is then used as the "leader." The other Micro-batches within that Mini-batch propagate the weights predicted by the "leader" during forward and backward propagation. To overcome the inconsistencies and stale weights caused by asynchronous Stage weight updates, this embodiment selects the first Micro-batch in each Mini-batch as the "leader" for weight prediction on any GPU of each computing node. This ensures that all Micro-batches within each Mini-batch use nearly consistent Stage weights during forward and backward propagation, effectively guaranteeing the learning efficiency of model parameters during pipelined parallel training. Figure 3 In (a), when T=4, the second Mini-batch contains Micro-batches 5-8, and Micro-batch 5 is selected as the leader. Figure 3 In (b), when T=2, the third one contains Micro-batch No. 5-6. Micro-batch No. 5 is selected as the leader. The significant feature of each leader is that for T Micro-batch from the same Mini-batch, the leader is always the first to perform forward and backward propagation in the pipeline. Figure 3 The solid black arrow in (a) depicts the process of predicting the weights of Micro-batch No. 5 as the leader of the second Mini-batch at T=4. Figure 3The black solid line arrow in (b) depicts the process of predicting the weights of Micro-batch No. 5 as the leader of the 3rd Mini-batch when T=2. Figure 3 (c) describes the process of the 5th Mini-batch predicting weights as its own “leader” when T=1. Figure 3 In (a), (b), and (c), all the solid black arrows start from the pipeline time unit where the "leader" performs forward or backward propagation, and point to the pipeline time unit corresponding to when the Mini-batch to which this "leader" belongs ends its lifecycle on GPU 0. The network weights predicted for the "leader" of the t-th Mini-batch are generated when T Micro-batches from the t-th Mini-batch sequentially enter the pipeline and prepare for forward and backward propagation. The leader is first generated through prediction. Then, the next T-1 micro-batches directly inherit the weights predicted by the leader for forward or backward propagation. Since all micro-batches in each mini-batch use nearly consistent model parameters during forward and backward propagation, the problem of stale weights is alleviated, effectively ensuring the effectiveness of model parameter learning during pipelined parallel training.

[0045] The weights for the first micro-batch prediction are calculated as follows:

[0046]

[0047] in, The weights generated by prediction, W t ΔW represents the current weight corresponding to the t-th pipeline time unit, where lr is the learning rate; s refers to the number of Stage weight updates between the current pipeline time unit and the pipeline unit corresponding to the Mini-batch containing the first Micro-batch after the end of backpropagation. t This represents the relative change of the current weight after one weight update. It is calculated based on the type of optimizer used in DNN training; the formula for calculating the number of consecutive updates, s, is:

[0048] During forward propagation,

[0049] During back propagation,

[0050] rank is the GPU ID, rank∈{0,1,…,N-1}, symbol The round() function rounds down to the nearest integer.

[0051] In this embodiment, the predicted weights always point to the pipeline time unit corresponding to the completion of the entire backpropagation of this Mini-batch on GPU 0. When each Micro-batch enters the pipeline sequentially and is ready to perform forward and backward propagation, the "leader" needs to use formula (1) to predict the weights. In formula (1), W t Since both lr and ΔW are known, the "leader" first needs to calculate ΔW when making weight predictions. t And s. In forward propagation, s is calculated using formula (2), while in backward propagation, s is calculated using formula (3). And ΔW t The gradient value can be calculated based on the type of optimizer used to train the DNN model. Assuming the "leader" completes backpropagation at time t, the gradient value... When using the momentum SGD optimizer to learn the model parameters, ΔW t =γ·ΔW t-1 +(1-γ)·g t .

[0052] For the Adam optimizer, ΔW t The calculation formula is:

[0053]

[0054] Furthermore, when learning DNN model parameters using optimizers such as RMSProp, AdaDelta, and AdaGrad, the prediction weights of the "leader" can also be easily calculated based on the mathematical expressions of these optimizers.

[0055] On each GPU, assume the "leader" has weights W when preparing to perform a forward propagation. t Then, the "leader" first calculates the number of updates s during forward propagation using formula (2), and calculates ΔW according to the type of optimizer used in DNN training. t Then the "leader" calculates the predicted future weights using formula (1). Finally, the "leader" uses the predicted future weights for forward propagation. It's important to note that the weights predicted by the "leader" during forward propagation need to be cached in the GPU. The next T-1 micro-batches can use the "leader's" predicted weights for forward propagation, thus avoiding duplicate weight predictions in other micro-batches. The weight prediction process during backpropagation is similar to that during forward propagation, except that formula (3) is used to calculate the number of updates s during backpropagation.

[0056] The training process for a Mini-batch is consistent across all compute nodes. For the i-th GPU on any compute node, as each Micro-batch within a Mini-batch completes backpropagation sequentially, the resulting gradient values ​​accumulate. On the i-th GPU across all compute nodes, after the last Micro-batch within the same Mini-batch completes backpropagation, all processes within the same communication group utilize the accumulated gradient values ​​and perform global gradient summation using the AllReduce communication operation method for distributed deep learning. Then, they update the weights of the corresponding Stage using the global gradient summation. The Stage updates its model parameters using the global gradient, and the update method is determined by the optimizer used.

[0057] Step 5: Training ends after all global batch data has been trained on the multi-machine, multi-card system.

[0058] This invention also provides a pipelined parallel training system for multi-GPU computing platforms. Given a multi-machine, multi-GPU computing system containing M GPUs, where M is the number of computing nodes and N is the number of GPUs on each computing node, the system includes the following modules:

[0059] Model splitting module: This module is used to vertically split the DNN model into stages corresponding to the number of GPUs on each computing node, load them onto the GPUs corresponding to each computing node, and then initialize the learning rate and optimizer parameters of the DNN model.

[0060] Input module: used to acquire a global batch data and divide the global batch data into M parts and distribute them to each computing node, so that each computing node obtains a batch data Mini-batch;

[0061] Parallel processing module: On each computing node, each batch of data Mini-batch is divided into T Micro-batch, which are then launched sequentially into a pipeline structure composed of GPUs. Each Micro-batch is arranged in the GPUs in an alternating manner of forward propagation and backward propagation.

[0062] Training termination module: Training ends when all global batch data has been trained on the multi-machine, multi-card system.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for pipelined parallel training of DNN models for multi-machine, multi-GPU computing systems, characterized in that, Includes the following steps: Step 1: Given a set containing A multi-machine, multi-GPU computing system, where M is the number of computing nodes and N is the number of GPUs on each computing node; Step 2: Vertically split the DNN model into segments corresponding to the number of GPUs on each computing node. Each stage is generated and loaded onto the GPU corresponding to each computing node. Then, the learning rate and optimizer parameters of the DNN model are initialized. Step 3: Obtain a global batch data Global Mini-batch, and divide the global batch data Global Mini-batch into M parts and distribute them to each computing node, so that each computing node gets one batch data Mini-batch; Step 4: On each computing node, divide each batch of data (Mini-batch) into T micro-batches, and sequentially transmit them into the network. In a pipeline structure composed of multiple GPUs, each micro-batch... The GPUs are arranged in a manner that alternates between one forward propagation and one backward propagation; for any computing node, the th... On all GPUs, after all Micro-batches in a Mini-batch have completed backpropagation, the first GPU on all compute nodes... Each GPU utilizes the accumulated gradient values ​​and performs global gradient summation through AllReduce, a communication operation method used for distributed deep learning, and then updates the weights of the corresponding Stage. Step 5: Training ends after all global batch data has been trained on the multi-machine, multi-card system; Each Mini-batch is divided into T Micro-batches, which use consistent prediction weights during forward and backward propagation. The weights of the consistent predictions are predicted by the first Micro-batch in each Mini-batch segment as the "leader", and the weights predicted by the "leader" are used by the other Micro-batch segments in the forward and backward propagation processes. In pipelined parallel training, each computing node is assigned Each process j pass j mod M Each operation corresponds one-to-one with the GPU on its compute node. The process is used to perform the forward and backward propagation calculations of the Stage and to perform global gradient summation within a communication group, which refers to all compute nodes... Processes with the same value form a communication group.

2. The method according to claim 1, characterized in that, The weights for the first micro-batch prediction are calculated as follows: in, These are the predicted weights. It is the current weight corresponding to the t-th pipeline time unit. lr The learning rate; s This refers to the number of Stage weight updates between the current pipeline time unit and the corresponding pipeline unit after the backpropagation of the Mini-batch containing the "leader" has ended. This represents the relative change in the current weight after one weight update.

3. The method according to claim 2, characterized in that, The process generates several auxiliary threads, which are used to transfer activation values ​​and gradient values ​​between GPUs, and each auxiliary thread is responsible for sending or receiving a tensor data.

4. The method according to claim 3, characterized in that, On any compute node, each process creates and manages a first-in-first-out queue (actQ) to store the activation value input by the Stage.

5. A pipelined parallel training system for multi-GPU computing platforms, characterized in that, Given a containing A multi-machine, multi-GPU computing system, where M is the number of computing nodes and N is the number of GPUs on each computing node, includes the following modules: Model partitioning module: Used to vertically partition the DNN model into segments corresponding to the number of GPUs on each computing node. Each stage is generated and loaded onto the GPU corresponding to each computing node. Then, the learning rate and optimizer parameters of the DNN model are initialized. Input module: used to acquire a global batch data and divide the global batch data into M parts and distribute them to each computing node, so that each computing node obtains a batch data Mini-batch; Parallel processing module: Used on each computing node to divide each batch of data (Mini-batch) into T micro-batches, which are then sequentially emitted into the processing module. In a pipeline structure composed of multiple GPUs, each micro-batch... The GPUs are arranged in a manner that alternates between one forward propagation and one backward propagation; for any computing node, the th... On all GPUs, after all Micro-batches in a Mini-batch have completed backpropagation, the first GPU on all compute nodes... Each GPU utilizes the accumulated gradient values ​​and performs global gradient summation through AllReduce, a communication operation method for distributed deep learning, and then updates the weights of the corresponding Stage; the T Micro-batches, which are divided from each Mini-batch, use consistent predicted weights during forward and backward propagation. The weights of the consistent predictions are predicted by the first Micro-batch in each Mini-batch segment as the "leader", and the weights predicted by the "leader" are used by the other Micro-batch segments in the forward and backward propagation processes. In pipelined parallel training, each computing node is assigned Each process j pass j mod M Each operation corresponds one-to-one with the GPU on its compute node. The process is used to perform the forward and backward propagation calculations of the Stage and to perform global gradient summation within a communication group, which refers to all compute nodes... Processes with the same value form a communication group; Training termination module: Training ends when all global batch data has been trained on the multi-machine, multi-card system.

Images