A neural network pipeline parallel training method for optimizing model division

Abstract

The application discloses a neural network pipeline parallel training method for optimizing model division. The neural network is modeled as a DAG graph, and the weight of the vertex and the edge in the graph is modeled by using the theoretical calculation time and the theoretical communication time of the model to construct a performance model. Through iterative search of the model division scheme, a solution set is obtained, and a model division scheme capable of minimizing the variance of the theoretical calculation time between the model slices and minimizing the global communication time is obtained. Under the model division scheme, the pipeline parallel technology is introduced to further speed up the model training process. By using the model division algorithm, the global communication time during neural network training is reduced while ensuring the load balance between the calculation devices.

Description

Technical Field

[0001] This invention belongs to the field of parallel training of neural networks, and specifically relates to a pipelined parallel training method for neural networks with optimized model partitioning. Background Technology

[0002] With the advent of the big data era, the amount of data generated in people's lives is constantly increasing. To process this massive amount of data, solve various complex problems, and meet users' demands for high accuracy in deep learning tasks, the number of layers in neural networks is continuously increasing. Large-scale neural network models significantly increase training time. Therefore, distributed parallel training methods are commonly used to accelerate model training.

[0003] Methods for distributed training of neural networks are generally classified into three categories. Data parallelism involves dividing a large amount of data into smaller batches and sending them to various computing devices for computation. After computation, parameters are updated through a certain communication method. Model parallelism involves dividing a large model into smaller models and deploying them on various computing devices for training. Hybrid parallelism combines data parallelism and model parallelism.

[0004] Data parallelism requires computing devices to have sufficient memory to hold the complete model, resulting in poor scalability. When partitioning the model in model parallelism, traditional partitioning algorithms do not comprehensively consider the load balancing of computing devices and overall communication time. Using model parallel training alone also results in low system throughput. Therefore, how to utilize specific distributed parallel training strategies for neural networks to ensure load balancing of computing tasks on hardware devices, minimize global communication time, and accelerate neural network training is a pressing issue in the field of neural network parallel training. Summary of the Invention

[0005] To overcome some shortcomings of existing technologies, this invention proposes a pipelined parallel training method for neural networks with optimized model partitioning, thereby accelerating the training process of neural networks.

[0006] The technical solution adopted in this invention is as follows:

[0007] This invention includes the following steps:

[0008] Step 1: Model the neural network model using a DAG graph, and set the weight of the vertices in the DAG graph to the theoretical computation time of the neural network layer, and the weight of the edges to the theoretical communication time between the neural network layers.

[0009] Step 2: Based on the DAG graph, the neural network model is partitioned to obtain a set of solutions. The best model partitioning scheme is obtained by using a model partitioning scheme evaluation algorithm. The partitioned model is then deployed on the corresponding GPU.

[0010] Step 3: Divide the input data into several micro-batches of equal size. After the first GPU completes the calculation of the current micro-batch, it passes the output to the next GPU. The next GPU continues to perform the forward propagation calculation of the neural network, while the first GPU begins the calculation of the next micro-batch.

[0011] Step 4: After all the micro-batch data has completed the forward and backward propagation calculations of the neural network, update the parameters of the neural network.

[0012] Step 5: Repeat steps 3-4 until the set number of rounds is reached to complete the training of the neural network.

[0013] To elaborate further, step 1 is as follows:

[0014] Step 1.1: Model the neural network using a DAG graph. The direction of the edges in the DAG graph represents the dependency relationship between layers of the neural network model.

[0015] Step 1.2: Assign weights to the vertices and edges in the original DAG graph. The weight of a vertex represents the theoretical computation time of a neural network model layer; the weight of an edge in the DAG graph represents the theoretical communication time between neural network model layers.

[0016] To elaborate further, step 2 is as follows:

[0017] Step 2.1, calculate the theoretical computation time tc for each layer of the neural network. k Given the number of GPUs m, each layer of the neural network is roughly evenly distributed across the GPUs, so that the sum of the theoretical computation time of the model layers allocated on each GPU is roughly equal, resulting in an initial model partitioning scheme, which serves as the initial solution set.

[0018] Step 2.2: Traverse the current solution set to obtain all movable model layers between model pieces for each solution, calculate the movement probability of the movable layers, and finally determine all movable layers based on the movement probability.

[0019] Step 2.3: Move all movable layers to obtain a solution set of model partitioning schemes, which will be used as the solution set for the next iteration. Add the solution set of the current iteration to the final solution set. Repeat step 2.2 until the specified number of iterations is reached.

[0020] Step 2.4: Obtain the final solution set, use the model to evaluate the merits of different solutions, and obtain the optimal solution.

[0021] Step 2.5: Divide the neural network model according to the optimal solution and deploy the model slices sequentially on the corresponding GPUs.

[0022] Furthermore, the algorithm for evaluating the merits of the model partitioning scheme is as follows: the variance of the model partitioning theoretical calculation time is used to evaluate the GPU load balancing, and the global communication time is used to measure the communication time required during model training. The merits of the model partitioning scheme are determined by comprehensively considering the two indicators.

[0023] To elaborate further, step 3 is as follows:

[0024] Step 3.1: Start training the model by dividing a data batch into several micro-batches and creating a thread for each GPU. This thread consists of an input queue (input_queue) and an output queue (output_queue).

[0025] Step 3.2 establishes dependencies between computational tasks to facilitate task scheduling. On the same GPU, dependencies are established for computational tasks on different micro-batches of data. On the same GPU, the next micro-batch begins computation only after the previous micro-batch is completed. Dependencies are also established for the same micro-batch of data executed on different GPUs. On different GPUs, the previous GPU completes the computation of its current micro-batch, places the output into its own `output_quene`, and then copies the data from `output_quene` to the next GPU.

[0026] The beneficial effects of this invention are:

[0027] This invention models the neural network as a DAG (Directed Acyclic Graph) and models the weights of vertices and edges in the graph using the model's theoretical computation time and theoretical communication time, thus constructing a performance model. Through iterative search of model partitioning schemes, a solution set is obtained, leading to a model partitioning scheme that minimizes the variance of theoretical computation time between model partitions and simultaneously minimizes global communication time. Under this model partitioning scheme, pipelined parallelism is introduced to further accelerate the model training process.

[0028] The model partitioning algorithm of this invention reduces the global communication time during neural network training while ensuring load balancing among computing devices. Compared to traditional model partitioning algorithms, the training time for one epoch is shorter under the model parallel strategy. Introducing pipeline parallelism reduces bubble time in the pipeline and lowers the total communication time, achieving better overlap between computation and communication, thereby further accelerating model training. Attached Figure Description

[0029] Figure 1 This is a flowchart of the parallel training of the neural network pipeline based on the optimized model partitioning of the present invention;

[0030] Figure 2These are the DAG graphs for neural network modeling in this invention, where (a) is an unweighted DAG graph and (b) is a weighted DAG graph.

[0031] Figure 3 This is a spatiotemporal graph of the neural network computation task of the present invention, wherein (a) is the forward propagation spatiotemporal graph and (b) is the backward propagation spatiotemporal graph;

[0032] Figure 4 This is a performance comparison chart of the model partitioning algorithm of this invention;

[0033] Figure 5 These are graphs showing the acceleration effect of the parallel algorithm of this invention with different micro-batch sizes;

[0034] Figure 6 The figures show a comparison of the parallel algorithm and single-GPU training accuracy of this invention, where (a) is the experiment with the ResNet-50 model and the CIFAR-10 dataset; (b) is the experiment with the ResNet-101 model and the CIFAR-100 dataset; and (c) is the experiment with the ResNet-152 model and the Caltech-101 dataset. Detailed Implementation

[0035] This embodiment discloses a pipelined parallel training method for neural networks with optimized model partitioning. The specific process is as follows: Figure 1 As shown.

[0036] It consists of three main parts: the first part is to model the DAG graph of the neural network to be trained; the second part is to divide the neural network model; and the third part is to perform pipelined parallel training.

[0037] The specific steps are shown in steps 1-3.

[0038] Step 1: Modeling a neural network DAG graph.

[0039] Step 1.1, define the DAG graph as G = (V, E), where the vertex set V represents the set of all layers in the neural network L = {l1, l2, ..., l...} n The edge set E represents all topological relationships between layers. For example... Figure 2 (a) shows an example of modeling.

[0040] Step 1.2: Abstract the theoretical computation time of each layer in the neural network into the weights of the vertices in the DAG graph. The theoretical computational cost of the convolutional layer is shown in Equation (1), the theoretical computational cost of the fully connected layer is shown in Equation (2), the theoretical computational cost of the BN layer is shown in Equation (3), and the theoretical computational cost of the pooling and ReLU layers is shown in Equation (4).

[0041]

[0042] f k =2×b×H k_in ×W k_in ×C k_in ×C k_0ut (2)

[0043] f k =2×b×C k_in ×H k_in ×W k_in (3)

[0044] f k =b×C k_in ×H k_in ×W k_in (4)

[0045] in C is the kernel size of the k-th layer. k_in H k_in W k_in Let C be the number of channels, height, and width of the input feature map for the k-th layer. k_out H k_out W k_out Let be the number of channels, height, and width of the output feature map of layer k. Let b be the batch size of the input.

[0046] Therefore, the theoretical computation time of the neural network model layer is shown in formula (5), where C is the theoretical computing power of the GPU and n is the total number of layers of the model.

[0047]

[0048] Among them tc k f is the theoretical computation time for a neural network layer. k Its floating-point computational cost (GFLOPs).

[0049] The communication time between layers is abstracted as the weights of the edges in the DAG graph. The output tensor size of the model layer is obtained from formula (6). Therefore, the theoretical communication time between the k-th layer and the (k+1)-th layer of the model is shown in formula (7), where B is the theoretical bandwidth of the GPU. Therefore, the weighted DAG graph is as follows: Figure 2 As shown in (b).

[0050] d k =b×C k_out ×H k_out ×W k_out (6)

[0051]

[0052] Where tsk d represents the theoretical communication time between a neural network layer and the next layer. k This represents the size of the tensor output by the neural network layer.

[0053] Step 2: Divide the neural network model.

[0054] Step 2.1, define the evaluation index of the model partitioning scheme. After partitioning the model, a sequence of model partitions is obtained: P = {P1, P2, ..., P...} m Each model slice sequence is a set of layers in the corresponding neural network, which is a continuous subset of the original model layers. The theoretical computation time Tc1, Tc2, ..., Tc of each model slice is obtained by summing the results according to formula (5). m The variance of the theoretical computation time between model segments is shown in formula (7), where Tc avg This represents the average theoretical computation time for model segmentation.

[0055]

[0056] The theoretical communication times Ts1, Ts2, ..., Ts of the model fragmentation are obtained according to formula (7). m-1 The calculation of global communication time is shown in formula (8).

[0057]

[0058] Step 2.2, Algorithm for evaluating the merits of model partitioning schemes. Assume there is a model partitioning scheme P... a The theoretical time variance σ of the corresponding model piecewise segment a Global communication time T comm_total_a Another model partitioning scheme P b The theoretical time variance σ of the corresponding model piecewise segment b Global communication time T comm_total_b .

[0059] If σ a and σ b If the difference is less than a given threshold ε, it indicates that the variances of the two partitioning schemes are similar, and the model partitioning scheme with the shorter global communication time is selected. If σ a and σ b If the difference is greater than a given threshold ε, it indicates that the variances differ significantly, and the model partitioning scheme with smaller variance should be selected.

[0060] Step 2.3: Generate the initial model partitioning scheme. First, the theoretical computation time list for each model layer is summed element-wise to obtain a list_flops containing the sum of the theoretical computation times of the current layer and all previous layers. Then, the values ​​in this list are normalized to eliminate the influence of singular time values, ensuring that the element values ​​are within the range [0, 1], resulting in a normalized list_flops_norm. Then, based on the number of GPUs m, a step size of 1 / m is obtained, and layers are sequentially partitioned into their corresponding model partitions. For example, elements with values ​​in [0, 0.25) are assigned to the first model partition, elements with values ​​in [0.25, 0.50) are assigned to the second model partition, and so on. This results in an initial model partitioning scheme where the theoretical computation times are roughly evenly distributed.

[0061] Step 2.4: Generate a solution set for the model partitioning scheme to obtain the optimal solution. Add the initial model partitioning scheme to the latest solution set. Traverse the solution set and determine all movable model layers for each solution. Except for the first and last layers of the original model, which cannot be moved, the first layer in a model partition can move forward to the end of the last layer of the previous model partition, and the last layer in a model partition can move backward to the beginning of the first layer of the next model partition. To reduce the search space, the movement probability of each layer is defined as r = ω. -t ω is a value greater than 1, and t is the number of times the corresponding layer has been moved, indicating that the more times a layer has been moved, the more likely it is to be discarded. Then, based on the layer moves, new solutions are generated, added to the new solution set, and the move count for that layer is incremented by 1. The new solution set serves as the solution set to be traversed in the next iteration. Finally, all newly generated, unique solutions from each layer iteration are collected to obtain a final solution set. Then, based on the model partitioning scheme evaluation algorithm in step 2.2, the optimal solution is obtained, which is the relatively optimal model partitioning scheme.

[0062] Step 3: Parallel training in a pipeline.

[0063] Step 3.1, Initialization. Create a thread for each GPU, responsible for an input queue (input_queue) and an output queue (output_queue). Place the corresponding computation tasks into the corresponding GPU and schedule them when the appropriate clock cycle begins.

[0064] Step 3.2: Divide the input batch of data into k equal micro-batches.

[0065] Step 3.3: Determine the space-time graph for pipeline parallelism. Define F. i,j For the forward propagation computation task of the i-th microbatch on the j-th GPU, B i,j This is the backpropagation computation task for the i-th microbatch on the j-th GPU. For example... Figure 3(a) shows the spatiotemporal diagram of the computational task during forward propagation. Figure 3 (b) shows the spatiotemporal diagram of the computational task during backpropagation. In this diagram, F is executed after clock 7 of the forward propagation. 4,4 After the computation task, backpropagation B is performed at clock 8. 4,4 The computational task.

[0066] Step 3.4: Establish dependencies between computational tasks. Taking forward propagation as an example, establish the computational task F for the current microbatch on the current GPU. i,j To the next micro-batch computation task F i+1,j This is a logical dependency. It establishes the computational task F of the current microbatch on the current GPU. i,j To the next computing task F on the GPU i,j+1 The dependency relationship is established by copying data from the current GPU's output queue to the next GPU. Backpropagation works similarly.

[0067] Step 3.5, scheduling process: At the beginning of each clock cycle, all tasks within that clock cycle are submitted to the input_queue of the corresponding GPU. The output is calculated and then submitted to the output_queue of that GPU.

[0068] Step 3.6: After all micro-batches have completed one full forward and backward propagation, calculate the loss and gradient, and update the parameters.

[0069] Step 3.7: Repeat steps 3.2-3.6 until the set number of rounds is reached, and output the trained neural network model.

[0070] Verification experiment:

[0071] (1) Experimental setup: The number of GPUs used was 4. Three combinations of neural network models and datasets were set up: ResNet-50 model and CIFAR-10 dataset, ResNet-101 model and CIFAR-100 dataset, and ResNet-152 model and Caltech-101 dataset.

[0072] (2) Data initialization: The image size of the CIFAR-10 and CIFAR-100 datasets is 32×32, and the image size of the Caltech-101 dataset is resized to 224×224.

[0073] (3) Model Partitioning Algorithm Performance Experiment: In three experimental groups, based on model parallelism, the optimized model partitioning algorithm proposed in this invention was compared with the partitioning algorithm based on theoretical computing power and the partitioning algorithm based on CUDA memory usage. The algorithm of this invention executes one training round faster than the other algorithms. Specifically, in the ResNet-152 and Caltech-101 experimental groups, the algorithm of this invention is 33.822s faster than the partitioning algorithm based on theoretical computing power and 18.063s faster than the partitioning algorithm based on CUDA memory usage. (Details are as follows...) Figure 4 As shown.

[0074] (4) Optimal Microbatch Size Experiment: In the three experimental groups, based on pipeline parallelism, the optimized model partitioning algorithm proposed in this paper was used to find the optimal microbatch size to achieve a smaller single-round training time. Experiments show that with the optimal microbatch size, the single-round training time is nearly half that of model parallelism. Specifically, as follows... Figure 5 As shown.

[0075] (5) Model Convergence Speed ​​Experiment: In the three experimental groups, pipelined parallel training and single-GPU model training under the optimal micro-batch size were compared. Compared with single-GPU training, pipelined parallel training achieved speedups of 1.19, 1.60, and 1.88 per epoch, respectively. Details are shown in the table below:

[0076] ResNet-50+CIFAR-10 30.123 25.316 ResNet-101+CIFAR-100 57.328 35.789 ResNet-152+Caltech-101 87.126 46.358

[0077] As the number of rounds increases, the model converges to approximately the same level of accuracy, specifically as follows: Figure 6 As shown in (a), (b), and (c).

Claims

1. A pipelined parallel training method for neural networks for image data processing, applied to a distributed computing system containing multiple GPUs, characterized in that... Includes the following steps: Step 1: Based on the theoretical computing power and memory bandwidth of each GPU, model the neural network model to be trained using a DAG graph, and set the weight of the vertices in the DAG graph to be the theoretical computing time of the neural network layer, and the weight of the edges to be the theoretical communication time between the neural network layers. Step 2: Based on the goals of GPU load balancing and minimizing global communication time, the neural network model is partitioned according to the DAG graph to obtain a set of solutions. The model partitioning scheme is evaluated by an algorithm to obtain a relatively optimal model partitioning scheme, and the partitioned model is deployed on the corresponding GPU. Step 3: Divide the input image data into several micro-batches of equal size. After the first GPU completes the calculation of the current micro-batch, it passes the output to the next GPU. The next GPU continues to perform the forward propagation calculation of the neural network, while the first GPU begins the calculation of the next micro-batch. Step 4: After all the micro-batch data has completed the forward and backward propagation calculations of the neural network, update the parameters of the neural network. Step 5: Repeat steps 3-4 until the set number of rounds is reached to complete the training of the neural network; Step 2 is as follows: Step 2.1, calculate the theoretical computation time for each layer of the neural network. Given the number of GPUs m, each layer of the neural network is roughly evenly distributed across the GPUs, so that the sum of the theoretical computation time of the model layers allocated on each GPU is roughly equal, resulting in an initial model partitioning scheme, which serves as the initial solution set. Step 2.2: Traverse the current solution set to obtain all movable model layers between model slices for each solution, calculate the movement probability of the movable layers, and finally determine all movable layers based on the movement probability. Step 2.3: Move all movable layers to obtain a solution set of model partitioning schemes, which will be used as the solution set for the next iteration. Add the solution set of the current iteration to the final solution set. Repeat step 2.2 until the specified number of iterations is reached. Step 2.4: Obtain the final solution set, use the model to evaluate the merits of different solutions, and obtain the optimal solution. Step 2.5: Divide the neural network model according to the optimal solution and deploy the model slices sequentially on the corresponding GPUs; The probability of movement is calculated using the following formula: Where r is the probability of movement. Let t be a number greater than 1, where t represents the number of times the corresponding model layer has been moved.

2. The pipelined parallel training method for neural networks for image data processing according to claim 1, characterized in that: Step 1 is as follows: Step 1.1: Model the neural network using a DAG graph. The direction of the edges in the DAG graph represents the dependency relationship between layers of the neural network model. Step 1.2: Assign weights to the vertices and edges of the original DAG graph; the weight of a vertex is the theoretical computation time of a neural network model layer, calculated using the following formula: in The theoretical computation time for a neural network layer. Here, n is the floating-point computation cost, n is the total number of layers in the neural network model, and C is the theoretical computing power of the GPU. The edge weights in a DAG (Directed Acyclic Graph) represent the theoretical communication time between layers of a neural network model, calculated using the following formula: in This represents the theoretical communication time between a neural network layer and the next layer. B is the size of the tensor output by the neural network layer, and B is the theoretical bandwidth of the GPU.

3. The pipelined parallel training method for neural networks for image data processing according to claim 1, characterized in that: The algorithm for evaluating the merits of the model partitioning scheme is as follows: the variance of the model partitioning theoretical calculation time is used to evaluate the GPU load balancing, and the global communication time is used to measure the communication time required during model training. The merits of the model partitioning scheme are determined by comprehensively considering the two indicators.

4. The pipelined parallel training method for neural networks for image data processing according to claim 3, characterized in that: If the difference between the variances of the theoretical calculation times of the two methods is less than a set threshold... Continue comparing the total communication times of the two solutions and select the one with the smaller value as the better solution; if the difference in the variance of the theoretical calculated times of the two solutions exceeds a set threshold... If the variance of the theoretical calculation time is smaller, then the solution with smaller variance is the better solution.

5. The pipelined parallel training method for neural networks for image data processing according to claim 1, characterized in that: Step 3 is as follows: Step 3.1: Start training the model by dividing a data batch into several micro-batches of equal size and creating a thread for each GPU. This thread consists of an input queue (input_queue) and an output queue (output_queue). Step 3.2: Establish dependencies between computation tasks to facilitate task scheduling; establish dependencies between computation tasks of different micro-batch data on the same GPU, so that the next micro-batch begins computation only after the previous micro-batch is completed on the same GPU; establish dependencies between the same micro-batch data executed on different GPUs; on different GPUs, the previous GPU completes the computation of the current micro-batch, puts the output into its own output_quene, and then copies the data of output_quene to the next GPU.

6. The method for pipelined parallel training of neural networks with optimized model partitioning according to claim 5, characterized in that: During the scheduling process: At the beginning of each clock cycle, all tasks within that clock cycle are submitted to the input queue of the corresponding GPU. The output is calculated and then submitted to the output queue of that GPU.

Images