A performance optimization method based on elastic average accelerating pipeline parallel training

Abstract

The application discloses a performance optimization method based on elastic average acceleration pipeline parallel training, and the method comprises the following steps: in the pipeline parallel training mode, integrating the elastic average technology, constructing a new training framework; by adjusting the scheduling strategy, avoiding the intermediate data from being accumulated in the display memory, thereby relieving the additional display memory occupation caused by the elastic average technology; the new pipeline parallel training framework has multiple adjustable parameters, and the historical information is counted to realize the automatic parameter optimization by the system.

Description

Technical Field

[0001] This invention belongs to the field of distributed deep learning, and in particular relates to a performance optimization method for pipelined parallel training in distributed deep learning systems. Background Technology

[0002] As application scenarios become increasingly complex, the size of deep neural network models continues to expand. Especially in the fields of natural language processing and recommendation systems, the size of these models can reach tens of gigabytes or more. This means that a single graphics card can no longer support the training of such models. Since traditional data-parallel training methods are no longer feasible, new model-parallel training methods have emerged in the industry. This method divides the model into multiple partitions, and each graphics card in the cluster only needs to be responsible for the training task of one partition.

[0003] However, basic model parallelism only utilizes one GPU at a time during training, resulting in very low resource utilization. To address this, previous work has proposed pipelined parallel training methods. For example... Figure 1 As shown, this method divides a batch of input data into multiple micro-batches, which are then piped through a cluster of GPUs. Compared to model parallelism, pipelined parallelism significantly improves resource utilization. However, because the backpropagation task depends on the result of the forward propagation task, upstream GPUs must wait for downstream GPUs to return computation results before starting their own backpropagation task. Therefore, existing pipelined parallelism still suffers from significant resource idle time, known as "bubbles." Furthermore, in pipelined parallelism, each GPU processes one micro-batch at a time. Due to the small size of the micro-batches, the arithmetic strength is insufficient, preventing training from fully utilizing the powerful computing capabilities of GPUs.

[0004] Two methods exist to reduce air bubbles, but both have drawbacks. The first method is to increase the batch size, such as... Figure 2 As shown. However, larger batch sizes also reduce training statistical efficiency, requiring more rounds of training jobs to reach the target accuracy. The second approach is to use a multi-version pipeline. For example... Figure 3 As shown, in GPU 1, the multi-version pipeline generates a model version after the backpropagation of micro-batch 1. Then, based on this version, the multi-version pipeline immediately performs the forward propagation of micro-batch 5 to eliminate bubbles. However, the drawback of this approach is that the version of micro-batch 1 cannot be released arbitrarily; it can only be released after the backpropagation task of micro-batch 5 is completed. This means that the system always maintains multiple model versions equivalent to the number of GPUs in the cluster, lacking scalability. Furthermore, the multi-version pipeline does not consider the impact of communication overhead between GPUs on performance in distributed scenarios.

[0005] In summary, existing distributed deep learning systems cannot fully utilize the computing power of clustered GPUs while supporting large-scale model training. Summary of the Invention

[0006] The purpose of this invention is to propose a performance optimization method for pipelined parallel training based on elastic averaging in a distributed deep learning system. This method integrates elastic averaging technology into the pipelined parallel training mode to construct a new training framework; by adjusting the scheduling strategy, it avoids intermediate data backlog in GPU memory, thereby alleviating the additional GPU memory occupation caused by elastic averaging technology; and by statistically analyzing historical information, it enables the system to automatically perform parameter tuning.

[0007] The elastic averaging technique refers to independently training multiple model replicas using different sampling data simultaneously during deep learning training to ensure statistical efficiency. Simultaneously, when updating model replicas, the update direction of a single replica is controlled by calculating the center point of multiple replicas, preventing excessive distances between replicas and ultimately avoiding excessively large training batches that reduce statistical efficiency. This invention integrates this technique into pipeline parallelism to support the use of multiple parallel pipelines during training, thereby improving training efficiency.

[0008] The scheduling strategy refers to the settings made by the training framework during the training process for the execution order and resource allocation of multiple tasks that can be executed immediately.

[0009] The specific technical solution for achieving the objective of this invention is as follows:

[0010] A performance optimization method for pipelined parallel training based on elastic averaging in a distributed deep learning system is disclosed. This method utilizes an elastic averaging strategy and a scheduling strategy to optimize and improve the training process in a pipelined parallel training mode. The method includes the following steps:

[0011] Step A: Divide the input model to be trained into multiple partitions according to the computational cost of each layer of the model, and distribute them to the various graphics cards in the cluster;

[0012] Step B: Create multiple parallel model partition copies and a reference model partition corresponding to each partition;

[0013] Step C: Divide each batch of input data into multiple micro-batches and process the multiple micro-batches of data sequentially in a pipeline manner;

[0014] Step D: Train and update multiple parallel models independently and in parallel during the processing of each batch of data, while using the elastic averaging method to dilute the weights of the parallel models and update them.

[0015] Step E: After each batch of data is processed, synchronize the training results from multiple parallel models to the reference model and update the reference model.

[0016] In this invention, multiple parallel models are used to probe the target parameters at different positions in the search space of model parameters. The reference model, as the central point of the parallel models, integrates all the probe results and is the final output model.

[0017] In step D of the model training and update process, a scheduling strategy is also applied to alleviate the additional memory usage caused by the elastic averaging technique. The step of adjusting the scheduling strategy includes:

[0018] Step D1: For the kth graphics card in the cluster, have it preprocess k-1 micro-batch forward propagation tasks in each training batch;

[0019] Step D2: After each batch of training is completed, dynamically increase the number of pre-processed micro-batches until the time taken to train a batch of data no longer decreases or the GPU memory space is insufficient.

[0020] The beneficial effects of this invention include:

[0021] First, this invention organically combines pipelined parallelism with elastic averaging, solving the problem of low resource utilization in the original pipelined parallelism. Second, this invention proposes a targeted scheduling strategy to alleviate the additional memory overhead caused by elastic averaging.

[0022] Compared to the original pipeline, this invention improves resource utilization by 86.1%. After applying a scheduling strategy, memory usage is reduced by 37.2%. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the basic execution process involving pipeline parallelism in existing technologies;

[0025] Figure 2 This is a schematic diagram of the basic execution process involving pipeline parallelism under larger training batches in existing technologies;

[0026] Figure 3 This is a schematic diagram of a multi-version execution process involving pipeline parallelism in existing technology;

[0027] Figure 4 This is a schematic diagram of an implementation method in the training framework of the present invention;

[0028] Figure 5 This is a schematic diagram of the second step in the scheduling strategy of the present invention.

[0029] Figure 6 This is a flowchart of the present invention. Detailed Implementation

[0030] The invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the invention are all common knowledge and general knowledge in the art, and the invention does not have any particular limitations.

[0031] This invention discloses a performance optimization method for pipelined parallel training based on elastic averaging. The method includes: integrating elastic averaging technology into the pipelined parallel training mode to construct a new training framework; avoiding intermediate data backlog in GPU memory by adjusting the scheduling strategy, thereby alleviating the additional GPU memory occupation caused by elastic averaging technology; and having multiple adjustable parameters in the new pipelined parallel training framework, which can be automatically tuned by the system by statistically analyzing historical information.

[0032] First, to improve resource utilization, this invention proposes a training framework based on elastic averaging. For example... Figure 4 As shown, the training process under this framework consists of four steps.

[0033] The first step is to divide a batch of input data into multiple micro-batches. The system processes multiple micro-batches of data in a pipeline manner, and performs forward and backward propagation tasks in parallel on multiple parallel models to obtain local updates.

[0034] The second step involves updating the parallel models. In addition to the local updates calculated in the first step, the system also applies elastic averaging to the parallel models. This ensures that the distance between the parallel models doesn't become too large while still allowing for the detection of the target model's parameters at different locations in the search space. Specifically, the weights of the parallel models are diluted with the weights of the reference model in a ratio of 1α:α, where α∈0,1 represents the dependence of the parallel models on the reference model. α=1 indicates that the parameters of the parallel models are completely updated to match the parameters of the reference model, reducing the detection efficiency in the search space. α=0 indicates that the parallel models are completely independent, free from the constraints of the reference model. In this case, the distance between the parallel models will increase, significantly reducing the overall statistical efficiency of training.

[0035] The third step is to send the partial update to the reference model. To prevent inter-process communication from blocking the pipeline, the system manages the sending work asynchronously through a message queue. That is, after the system puts the partial update into the message queue, it will immediately start the next batch of parallel model training tasks. A separate process will listen to the message queue and send the data in it to the reference model.

[0036] The fourth step is to accumulate all the local updates received from the parallel models and update the reference model after each batch is completed.

[0037] The above describes the specific implementation process of building a training framework based on elastic averaging. In a distributed deep learning system, this method can be implemented using the relevant code from Method 1 and Method 2, as shown below:

[0038]

[0039]

[0040]

[0041] Secondly, since the system needs to maintain multiple parallel models in the new training framework, there is additional GPU memory usage. To address this drawback, this invention designs a new scheduling strategy that avoids intermediate data accumulating in GPU memory by adjusting the execution order of training tasks, thereby saving storage space. This strategy consists of two steps.

[0042] The first step, to minimize GPU memory usage, is to alternately execute forward and backward propagation tasks as much as possible at the start of training. That is, after completing a micro-batch of forward propagation, the system will execute the next micro-batch of backward propagation to avoid accumulating forward propagation results. Specifically, if there are K GPUs in the cluster, the k-th GPU can only execute K-k+1 micro-batch forward propagation tasks in advance during each training batch. After that, the first micro-batch of backward propagation tasks may arrive at any time. Executing K-k+2 micro-batch forward propagation tasks in advance could increase the amount of data accumulating in GPU memory.

[0043] The second step involves dynamically increasing the number of pre-processed micro-batches. The method of alternating between forward and backward propagation tasks in the first step has performance limitations in a distributed environment. For example... Figure 5 As shown, compared to the original scheduling, this method will significantly reduce performance when communication overhead exists. Therefore, at the end of each batch, when GPU memory is sufficient and training performance is not yet saturated, the system will increase the number of pre-processed micro-batches by 1.

[0044] Specifically, Figure 5In (a), the original scheduler completes all forward propagation tasks first, and then all backward propagation tasks during the training of a batch of data. The advantage of this approach is that it can fully utilize the advantages of pipelined computing and communication time overlap, maximizing training efficiency. However, the disadvantage is that when the first GPU completes the forward propagation task of the 8th micro-batch, it needs to maintain the intermediate data of the entire batch in GPU memory, resulting in excessively high peak GPU memory usage.

[0045] Figure 5 In (b), alternating scheduling maintains alternating execution of forward and backward propagation tasks for different micro-batches during the training of a batch of data. Therefore, the first graphics card only needs to maintain intermediate data for a maximum of two micro-batches in its memory. However, the drawback is that the computation and communication time cannot be completely overlapped, sacrificing training efficiency.

[0046] Figure 5 In (c), the new scheduling combines the advantages of both the original scheduling and the alternating scheduling, while avoiding their shortcomings. Specifically, compared to the original scheduling, under the new scheduling, the first graphics card begins to alternately execute forward and backward propagation tasks after the third micro-batch of forward propagation tasks. Therefore, its peak memory usage is lower than that under the original scheduling. Compared to alternating scheduling, the first GPU in the new scheduling performs a micro-batch forward propagation task in advance, which allows the system to overlap computation and communication time in subsequent training tasks, solving the training efficiency problem of alternating scheduling and achieving the same training time as the original scheduling.

[0047] Finally, the overall framework structure of this invention is divided into three layers, as follows: Figure 6 As shown, the first layer partitions the input model structure and allocates it one-to-one to the GPUs in the cluster. The distributed model training task is further scheduled by the second-layer scheduler, which uses the scheduling strategy of this invention to determine the execution order of the training tasks based on the user-given memory limits and performance data collected during subsequent runs, and then passes them to the next layer. The third-layer runtime is responsible for the actual operation, including receiving input data, executing training tasks, and providing performance statistics to the upper-layer scheduler.

[0048] The above describes the specific implementation process of parallel training based on elastic averaging in a distributed deep learning system.

[0049] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of the inventive concept are included in this invention and are protected by the appended claims.

Claims

1. A performance optimization method based on elastic averaging accelerated pipeline parallel training, characterized in that, The method includes the following steps: Step A: Divide the input model to be trained into multiple partitions according to the computational cost of each layer of the model, and distribute them to the various graphics cards in the cluster; Step B: Create multiple parallel model partition copies and a reference model partition corresponding to each partition; Step C: Divide each batch of input data into multiple micro-batches and process the multiple micro-batches of data sequentially in a pipeline manner; Step D: Train and update multiple parallel models independently and in parallel during the processing of each batch of data, while using the elastic averaging method to dilute the weights of the parallel models and update them. In step D of the model training and update process, a scheduling strategy is also applied to alleviate the additional memory usage caused by the elastic averaging technique. The steps of the scheduling strategy include: Step D1: For the kth graphics card in the cluster, have it preprocess k-1 micro-batch forward propagation tasks in each training batch; Step D2: After each batch of training is completed, dynamically increase the number of pre-processed micro-batches until the time taken to train a batch of data no longer decreases or the video memory space is insufficient. In step D, the weights of the parallel model are diluted with the weights of the reference model by a proportion of 1 / 3. ,in This represents the dependence of the parallel model on the reference model. This means that the parameters of the parallel model are completely updated to those of the reference model, which reduces the detection efficiency in the search space. This indicates that the parallel model is completely independent and free from the constraints of the reference model; Step E: After each batch of data is processed, synchronize the training results from multiple parallel models to the reference model and update the reference model.

2. The method as described in claim 1, characterized in that, In step C, the input data of a batch is divided into multiple micro-batches, and the multiple micro-batches are processed in a pipeline manner. Forward and backward propagation tasks are performed in parallel on multiple parallel models to obtain local updates.

3. The method as described in claim 1, characterized in that, In step E, the sending work is managed asynchronously through a message queue. After the system puts the local update into the message queue, it will immediately start the next batch of parallel model training tasks. A separate process listens to the message queue and sends the data in it to the reference model.

Images