A parallel training method based on a hybrid parallel deep convolutional neural network

Abstract

The application provides a mixed parallel-based deep convolutional neural network parallel training method for quickly training a large model with a large data set. The method comprises in-group parallelism and inter-group parallelism. For in-group parallelism, firstly, an automatic model partitioning method is used to divide the model into multiple partitions, and the multiple partitions are placed on different GPUs respectively. Then, the small batch data after partitioning is sent to different GPUs one by one in succession, and the multiple partitions are trained in a pipeline parallel manner. Inter-group parallelism replicates model copies on each group, independently calculates gradients, and then updates the summary of the gradients on the inter-group partition to maximize the overlap of calculation and communication, thereby reducing the gradient synchronization time. Experimental results show that compared with the traditional data parallelism and the most advanced parallel training torchgpipe method, the acceleration performance of the method provided by the application is obviously improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of high-performance computing, and particularly relates to a deep convolutional neural network parallel training method based on hybrid parallelism. BACKGROUND

[0002] Deep Convolutional Neural Network (DCNN) plays a crucial role in image classification, autonomous driving, intelligent medical care and intelligent agriculture due to its complex model and large sample support. For example, Baidu Apollo can intelligently perceive complex road scenes to provide autonomous driving services, and Cox model can accurately predict the risk of COVID-19 patients developing critical illness according to clinical characteristics. In addition, it also accounts for a large proportion in other application fields such as sentiment analysis, electronic health assistants, national defense, game entertainment, etc.

[0003] With the increase of training data size and the increasing complexity of deep convolutional neural network model, under the driving of massive data, deep convolutional neural network containing more parameters is designed to solve more complex tasks. The increase of training data and the complexity of model structure involve a large number of parameter calculations, which leads to the problem of too long training time. Parallelization has become an important technical means to accelerate neural network training. The traditional distributed deep convolutional neural network parallel training method is mainly divided into the following three categories: 1) data parallel. Data parallelism is a means of data division, training the entire model on each node, and synchronizing parameters by parameter server after each iteration, and then broadcasting the parameters for the next iteration. The data parallel method is simple and effective, but it has the disadvantages of high GPU memory consumption, expansion bottleneck and inability to achieve global optimal solution. 2) Model parallel. Model parallelism is to divide a large model into disjoint sub-models, and distribute the sub-models to each sub-node for training or distribute different parameters between layers to different hardware. The traditional model parallelism will bring a large communication and synchronization overhead. 3) Hybrid parallel. Hybrid parallelism is a mechanism that uses data and model parallelism for acceleration. This strategy uses a new runtime scheduling method to ensure model convergence while improving computational efficiency. Although this method has achieved certain results, there is still a lot of parallelism in the method runtime, and compared with the theoretical optimal performance, there is still a lot of room for improvement. Therefore, how to parallelize the training process of deep convolutional neural network model, how to shorten the training time of deep convolutional neural network model, and then shorten the running time of the field using deep convolutional neural network model, has become a big problem in the field of deep learning. SUMMARY

[0004] The present application aims to overcome the above-mentioned defects of the prior art, and provides a deep convolutional neural network parallel training method based on hybrid parallelism to solve the problem of long training time of the deep convolutional neural network model in the prior art.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] A deep convolutional neural network parallel training method based on hybrid parallelism comprises the following steps:

[0007] Step 1, input data, divide the data into several small batches of data;

[0008] Step 2, divide the small batch of data into m micro batches of data, and input each micro batch of data into a group;

[0009] Step 3, sequentially perform forward calculation and reverse calculation on the micro batch of data;

[0010] Each previous partition GPU in each group performs forward calculation on the previous micro batch of data, transmits the calculation result to the next partition GPU, the next partition GPU continues forward calculation according to the calculation result of the previous partition GPU, and the previous partition GPU performs forward calculation on the next micro batch of data;

[0011] After each micro batch of data is calculated by all GPUs in the group, reverse calculation is performed by all GPUs, each previous partition GPU in each group performs reverse calculation on the previous micro batch of data, transmits the calculation result to the next partition GPU, the next partition GPU continues reverse calculation according to the calculation result of the previous partition GPU, and the previous partition GPU performs reverse calculation on the next micro batch of data, and each partition GPU obtains the corresponding gradient through reverse calculation;

[0012] Forward calculation and reverse calculation are performed synchronously until all micro batches of data in the small batch of data corresponding to the group are calculated; k partition GPUs in a group are calculated synchronously; N groups are calculated synchronously to calculate the small batch of data corresponding to each group; one partition GPU corresponds to one area in the deep convolutional neural network, one GPU is in one partition GPU, and one area in the deep convolutional neural network comprises a plurality of network layers in the neural network;

[0013] Step 4, after all the same area GPUs in different groups complete calculation, the same area GPUs in different groups perform gradient synchronization;

[0014] Step 5, after the gradient synchronization of all the same layer GPUs is completed, steps 2-4 are repeated until all data is calculated.

[0015] The present application is further improved in that:

[0016] Preferably, the determination process of k-partition GPUs in a group is as follows:

[0017] (1) Calculate the execution time of each network layer in the deep convolutional neural network based on small batch data;

[0018] (2) Normalize the execution time of each layer;

[0019] (3) Randomly and equally allocate a number of network layers in the deep convolutional neural network to each of the k-partition GPUs in a group;

[0020] (4) Obtain the running computation time of each partition, and adjust and exchange the network layers in each partition by a double-loop method;

[0021] (5) Take the running computation time of each partition as a reference, and end step (4) when the difference between the partition P max with the largest execution time sum and the partition P min with the smallest execution time sum is less than a threshold value, and obtain the final k-partition GPUs;

[0022] (6) Input the results of the k-partitions to the GPU.

[0023] Preferably, the double-loop method includes an inner loop and an outer loop, the outer loop is used to find the largest partition P max , and the inner loop is used to find the smallest partition P min .

[0024] Preferably, in step (5), the threshold value is the largest execution time of the model network layer.

[0025] Preferably, in step (2), the normalization calculation formula is:

[0026]

[0027] Wherein, t i is the execution time of each network layer, min(T) represents the network model layer with the shortest execution time, and max(T) represents the network model layer with the longest execution time.

[0028] Preferably, in step 3, each partition GPU has a work thread, and each work thread includes a pair of task columns:

[0029] in_queue i , out_queue i , i = 1, 2, …, k

[0030] The in_queue iin_queue for input queue, out_queue i out_queue for output queue.

[0031] Preferably, in step 3, the process of transferring the result of the previous partition GPU computing the micro-batch data to the next partition GPU is:

[0032] copying the output of F in the out_queue of the j-1 layer GPU j-1 i,j-1 to the in_queue of the j layer GPU, and putting the result of the j layer GPU computing into the out_queue j j .

[0033] Preferably, in step 3, the loss function of each micro-batch data is calculated simultaneously, and the back propagation is calculated according to the loss function.

[0034] Preferably, in step 4, each partition is provided with a hook function, and the gradient synchronization is performed on different groups of the same GPU through the hook function.

[0035] Compared with the prior art, the present application has the following beneficial effects:

[0036] The present application provides a deep convolutional neural network parallel training method based on hybrid parallelism, which is used for quickly training large models with large data sets. The method includes intra-group parallelism and inter-group parallelism. For intra-group parallelism, first, an automatic model partitioning method is used to divide the model into multiple partitions, and the multiple partitions are placed on different GPUs. Then, the divided micro-batch data is sent to different GPUs one after another in a pipeline parallel manner, and multiple partitions are trained. Inter-group parallelism replicates model copies on each group, independently calculates gradients, and then updates the summary of these gradients on the inter-group partition to maximize the overlap of computation and communication, thereby reducing the gradient synchronization time. Experimental results show that compared with traditional data parallelism and the most advanced parallel training torchgpipe method, the acceleration performance of the method proposed in the present application is significantly improved. When the training result reaches the same Top-1 accuracy, the speedup ratio is improved by 59.6% and 14.3% compared with the above two methods. This shows that the method can significantly improve the speedup performance of training on large models with large data sets. Therefore, the method proposed in the present application has broad application prospects in the field of deep learning parallelization.

[0037] ​​Further, the method of the present application consists of two parts: intra-group parallelism and inter-group parallelism. Intra-group parallelism involves automatically splitting a large DCNN model into multiple partitions, which are trained in a pipeline manner in parallel, thereby minimizing the training time of large models. Inter-group parallelism adopts a data parallel method based on a computation and communication overlap strategy to significantly improve the training parallelism, and uses a partition-based overlap computation and communication method in the intra-group pipeline model parallel of inter-group partition to reduce the communication time overhead. The experimental results of this method show that this method can significantly improve the speedup performance of training on large models with large datasets.

[0038] Further, the method of the present application includes intra-group parallelism and inter-group parallelism. For intra-group parallelism, the model is first divided into multiple partitions using an automatic model partitioning method, and the multiple partitions are placed on different GPUs. Then, the mini-batch data after partitioning is sent to different GPUs one after another in a continuous manner, and the multiple partitions are trained in a pipeline parallel manner. Inter-group parallelism replicates model copies on each group, independently calculates gradients, and then updates the summary of these gradients on inter-group partitions to maximize the overlap of computation and communication, thereby reducing the gradient synchronization time. The present application is used for fast training of large models with high parallelism, strong scalability and high GPU utilization.

[0039] Further, the present application proposes an intra-group pipeline model parallel method based on automatic model partitioning for large model training and improving GPU utilization. First, the proposed automatic model partitioning method is used to partition a large model in a group to obtain the best partitioning result. Then, the mini-batch is divided into multiple micro-batch data, and the intra-group pipeline model parallel method is used to send them to the partitions. This method can effectively allow the pipeline and parallel training of large models without manually partitioning the model.

[0040] Further, the present application designs an inter-group data parallel method based on computation and communication overlap to improve the training parallelism. First, each group loads a copy of the entire large model for parallel training in inter-group data parallelism. Then, the time of overlapping computation and communication is maximized during backpropagation to reduce the time-consuming of gradient synchronization, and finally the global weights of the neural network are updated. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The mixed parallel training architecture diagram of the present application;

[0042] Figure 2 The model automatic partitioning flowchart of the present application;

[0043] Figure 3A group-in pipeline model parallel execution graph for the application;

[0044] Figure 4 An inter-group data parallel graph based on computation and communication overlap for the application;

[0045] Figure 5 A computation and communication overlap performance improvement result graph for the application.

[0046] Figure 6 A performance improvement result graph of the method proposed by the application and two parallel training methods;

[0047] Figure 7 A leaf disease data set of apples in the embodiment;

[0048] Wherein (a) is spot anthracnose; (b) is brown spot; (c) is mosaic; (d) is gray spot; (e) is iron spot. DETAILED DESCRIPTION

[0049] The application will be further described in detail below with reference to the accompanying drawings:

[0050] In the description of the application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the application and simplifying the description, and therefore cannot be understood as indicating or implying that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the application; the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance; in addition, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connection" should be understood broadly, for example, it can be fixed connection, or detachable connection; it can be directly connected, or indirectly connected through an intermediate medium; it can be the communication between two elements. For those skilled in the art, the specific meaning of the above terms in the application can be understood according to the specific circumstances.

[0051] The application discloses a deep convolutional neural network parallel training method based on hybrid parallelism, which proposes a new hybrid parallel training architecture (as shown in Figure 1As shown in the figure, it mainly consists of two parts: intra-group parallelism and inter-group parallelism. Intra-group parallelism includes the model automatic partitioning method and the intra-group pipeline model parallelism method. Inter-group parallelism proposes a data parallelism method based on the overlap of computation and communication. First, the DCNN model is divided by the model automatic partitioning method, and the optimal model partitioning result is obtained by iterative calculation. The divided model parameters are deployed on the GPU devices in each group. Then each group divides the data to obtain small batches of data after division, and the divided small batches of data are continuously sent to different GPUs one by one, and multiple partitions are trained in parallel based on the intra-group pipeline model parallelism method. Finally, the GPUs within each group perform gradient synchronization and global parameter updates based on the data parallelism method of computation and communication overlap. Due to the overlap of subtasks in the pipeline parallel method and the data parallel method of computation and communication overlap, the efficiency of executing tasks in the same time is increased, the total execution time is shortened, and the acceleration ratio is significantly improved.

[0052] The parallel training method for deep convolutional neural networks based on hybrid parallelism consists of four main steps. The first step is resource allocation. This step lays the foundation for model parallel training. The remaining three steps are: automatic model partitioning, intra-group pipeline model parallelism, and data parallelism based on overlapping computation and communication. After completing these three steps, the trained model parameters are finally obtained through continuous iterative calculation. Figure 2 , the method specifically comprises the following steps:

[0053] Step 1: Resource Configuration

[0054] Figure 1 The architecture of the proposed method is shown. Resource allocation is the process of determining relevant parameters based on demand at the beginning of training. Based on the single-machine multi-GPU experimental environment and the actual amount of model parameters that need to be processed, the number of groups N is determined (the principle of division is that the number of groups can meet the running speed requirements of the entire process and the cost is low), the number of graphics processors (GPUs) k required in each group, and the number of micro-batches m in the pipeline parallel method are determined. Then, according to resource allocation, computing resources are allocated to N groups, each group consisting of k homogeneous GPUs (for example, k = 4, such as Figure 1 Next, given a large DCNN model and k allocated GPUs, the automatic model partitioning method divides the model into k partitions, maximizing the acceleration of the pipeline execution within each group. This operation plays a crucial role in the entire training process. Deep convolutional neural network training is a time-consuming process, typically involving a large number of computations. Therefore, combining resource allocation with parallelization of network training can effectively accelerate the entire model training process, fully utilizing GPU resources to accelerate the training of large DCNN models.

[0055] Step 2: Automatic model division

[0056] like Figure 2 As shown in the automatic model partitioning flow chart, automatic model partitioning consists of five main steps. Given a large DCNN model and resource configuration parameters, the automatic model partitioning method divides the model into k partitions and places them on k GPU devices within a group. This maximizes the acceleration performance of the pipeline execution within each group. At the same time, the execution time of each partition is kept as equal as possible to maximize load balancing. The specific steps of automatic model partitioning are as follows:

[0057] Step 2.1 Input model parameters and resource configuration

[0058] Specifically, based on the results of the resource allocation process in step 1, we determine the number of groupings N, the number of GPUs used within each group k (corresponding to the number of partitions), and the number of micro-batches m. The input to the automatic model partitioning method is an L-layer DCNN model, a mini-batch of data, and the number of partitions k. The output is a list of the number of layers in each partition of the DCNN model. The mini-batch is the specific data to be processed, such as an image to be processed. It is sourced from all the data to be processed, and a portion of the data to be processed is used as the mini-batch data to build the model.

[0059] Step 2.2 Collect the execution time of each layer of the network model

[0060] The computational execution time T = [t1, t2, ..., t L ], where t i It represents the execution time of each layer of the network, which includes the sum of the forward calculation and the back propagation calculation time. The subscript L represents the total number of layers of the model.

[0061] Step 2.3 Normalize the execution time of each layer

[0062] When the levels of various indicators differ greatly, if the original collected execution time is used directly for analysis, the role of the higher values ​​in the comprehensive analysis will be highlighted, and the role of the lower values ​​will be relatively weakened. Therefore, in order to ensure the reliability of the results, the original execution time values ​​need to be standardized. Use the normalization formula (1) to convert the execution time t of each layer of the network into i Normalized to the range [0,1]. Min(T) represents the network model layer with the shortest execution time, and max(T) represents the network model layer with the longest execution time.

[0063]

[0064] Step 2.4 Initialize the partition

[0065] Initialize the partition according to T in step 2.2 and k in step 2.1, directly divide the GPUs in a group, average the network layers of several neural networks in one GPU, and get the initialization result s, where each partition result uses formula (2) and formula (3). i is the number of layers in each partition of T after initialization partitioning. Each group corresponds to a mini-batch of data, and different groups correspond to different mini-batches of data.

[0066] p i =[t (i-1)*s+1 ,t (i-1)*s+2 ,...,t i*s ] (2)

[0067]

[0068] Step 2.5 Iterative analysis of the best partitioning results of the model

[0069] According to the initialization partition result P=[p1,p2,...,p k ], the optimal partitioning result is found through the double loop in the model automatic partitioning algorithm. In each iteration, the outermost loop is used to find the partition P with the largest sum of time. max The inner loop first finds the partition P with the lowest time sum min , if the partition P with the largest sum of time max The partition P with the smallest sum of time min When the difference is less than a threshold (this value is the maximum execution time in the model network layer), the algorithm outputs the best partitioning result (returns a partitioning result as shown in formula (5)) and terminates the loop. The termination condition is shown in formula (4). In contrast, P max and P min When the difference is greater than or equal to this threshold, continue to adjust the partition division, and the adjusted partition P will become P', P h It is the adjacent partition of the minimum partition and is located in the maximum partition P max and the minimum partition P min When max <min时,证明P max In P min On the left side, adjust h = min-1, and at the same time, h The last element in the partition is moved to P min In the partition, when max>min, P max In P min On the right side, adjust h=min+1, and change Ph The first element in the partition moves to P min In the partition, if when max == h, break the inner loop. The total time complexity of this method is only O(k*L 3 ) (where k is the number of partitions, and L is the number of layers).

[0070] sum(p max )-sum(p min )<=max(t i ) (4)

[0071] [len(p1),len(p2),...,len(p k )] (5)

[0072] Step 3, intra-group pipeline model parallel

[0073] As shown in Figure 3 , the intra-group pipeline model parallel method is to divide each group of small batch data into m micro batch data according to the partition result generated in step 2, and then send these micro batch data to multiple partition GPUs for calculation. In addition, these partition GPUs train in a pipelined parallel manner within a group. The basic idea is that after the previous partition GPU calculates a micro batch data, the calculation result of the micro batch data is copied and transmitted to the next partition GPU, and multiple partitions are calculated at the same time. The above divided sub-models are trained at the same stage. Secondly, the large small batch is divided into small batches. After the first micro batch is processed in the first stage, the activation of this stage is immediately passed to the next stage. At the same time, the second micro batch enters the first stage, so as to make use of the idle time in pure model parallel. The specific steps of intra-group pipeline model parallel are as follows:

[0074] Step 3.1 parameter input

[0075] Specifically, according to the partition division result in the best division process of the model in step 2.5 iteration analysis, the partition division list [len(p1),len(p2),...,len(p k )] is obtained, and the small batch data in the model parameter in 2.1 is taken as the input data. The parameter input includes the partition division result generated based on the model automatic division and the small batch data in the model parameter.

[0076] Step 3.2 initialization related work

[0077] This step mainly includes two aspects of work: creating work threads and copying the partition division result in step 2.5 to the GPU device. First, for the number of partitions k in step 2.1, create k work threads Th1, Th2,..., Th k, each worker thread Th i There is a pair of task queues (in_queue i ,out_queue i ,i=1,2,...,k). Each thread is responsible for scheduling and executing the data submitted to the corresponding in_queue i Tasks in the queue. Secondly, the multiple partitions of the divided model need to be copied to the corresponding GPU devices.

[0078] Step 3.3 Pipeline Scheduling Method

[0079] Step 3.3.1 Task scheduling. Figure 3 All micro-batch data F i,j (represents the forward computing task, where j represents the partition number and i represents the micro-batch number). i,j The task is submitted to in_queue every clock cycle i , so that all tasks can be processed asynchronously. All tasks are processed accordingly and the corresponding results are put into the corresponding out_queue i (i=1,2,..,k).

[0080] Step 3.3.2 Data dependency processing. Since the deep learning framework PyTorch is a dynamic computational graph, PyTorch's automatic differentiation (autograd) engine causes the backpropagation process to not run in the reverse order of forward computation. Therefore, two key dependency structures of the graph must be constructed to force the pipeline to work in parallel as expected. See Figure 3 , first we need to build virtual dependencies, the micro-batch data F in the ijth GPU i,j Depends on the micro-batch data F in the (i-1)jth GPU i-1,j The calculation results, F i-1,j Must be in F i,j Completed before execution. After each micro-batch data is forward-calculated by all GPUs, it starts to be backward-calculated. The result of each GPU's backward calculation is the gradient of the GPU in that partition. Similar to the forward calculation process, the backward propagation task B i,j Must be in B i-1,j Completed before execution, the second step is to build the replication dependency. Therefore, the forward and reverse calculations in each group are performed synchronously. Corresponding to the relevant content in step 3.3.1 task scheduling, copy from out_queue j-1 The obtained F i,j-1 Output to device j On the GPU corresponding to the device. Submit F i,j to in_queue jThe calculated result is put into the out_queue j Through task scheduling and data dependency processing, the data can be trained in a pipeline manner, thereby accelerating the training speed.

[0081] Step 4, inter-group data parallel method based on computation and communication overlap

[0082] In order to further greatly improve the parallelism to accelerate the training of large DCNN models, the present application proposes an inter-group data parallel method based on computation and communication overlap. This method has two key steps. First, it breaks the limitation of traditional data parallelism, that is, the DCNN model can only be loaded into the memory of a single GPU. The inter-group data parallelism is supported by the concept of a group consisting of k GPUs. When a single GPU has limited resources, the group is allowed to load a large DCNN model by aggregating multiple GPUs. There are multiple GPUs in each group to load the entire DCNN model. The training data set is divided into small batches, and forward propagation is performed in pipeline model parallelism to calculate the loss function. Second, backward propagation is performed in each group based on the computation and communication overlap strategy to calculate the gradient, and then gradient synchronization is performed. Finally, the model parameters are updated. Because each group starts from the same copy of the model, and all gradients of each group are reduced, the weights between groups are the same. The specific steps are as follows:

[0083] Step 4.1 partition-based overlap computation

[0084] As Figure 4 shown, the present application proposes a partition-based overlap computation, and communication is performed on the inter-group partition to reduce the gradient synchronization time. Considering that the all-reduce collective communication is a time-consuming operation, especially on large models with a large number of parameters, the time overhead can be effectively reduced by reducing the number of communications. As Figure 5 shown (a simple solution that inserts a gradient synchronization phase after backward propagation and before updating parameters). Unlike the above simple solution, the partition-based overlap computation further achieves higher throughput and lower latency by integrating all the gradients of the model parallel partition into one all-reduce communication operation, instead of starting an all-reduce communication operation for each gradient tensor after backward propagation. The specific steps of the partition-based overlap computation are as follows:

[0085] Step 4.1.1 partition registration hook function

[0086] This step registers an autograd hook function for each partition in all groups (PyTorch provides a mechanism for autograd to save tensors) to start an all-reduce communication operation. The hook in PyTorch is equivalent to a plug-in. Some additional functions can be implemented without modifying the main code. Corresponding to PyTorch, the main body is the forward calculation and back propagation process, and the additional function is to operate on the variables of the model, such as "extracting" the feature map, "extracting" the gradient of the non-leaf tensor, modifying the tensor gradient, and so on.

[0087] Step 4.1.2 starts intra-group parallel and inter-group parallel training

[0088] When the hook function registration in step 4.1.1 is completed, the intra-group pipelined model parallel in step 3 and the inter-group data parallel training based on the overlap of calculation and communication in step 4 are started.

[0089] Step 4.1.3 gradient synchronization

[0090] First, after each group completes the back propagation calculation of the current small batch, the corresponding hook function is triggered. Then, an all-reduce communication operation is started asynchronously to synchronize all gradients of the corresponding partition GPUs in all groups. Specifically, the gradient is a vector, which is the partial derivative of a function with respect to a variable. The gradient points to the direction in which the function value at each point decreases the most. For example, as shown in Figure 4 , N = 2 (two groups) and k = 4 (each group contains 4 GPUs). Among them, GPU4 and GPU8 correspond to the same model partition (P4). When the back propagation calculation task of the last partition P4 in the two groups is completed, an all-reduce communication operation is started to synchronize the gradients of the partition P4 in all groups. At this time, GPU3 continues to perform the B 1,3 propagation calculation task on the P3 partition. This forms an overlap between P4 partition gradient synchronization and back propagation calculation, so the proposed calculation and communication overlap strategy can start an all-reduce communication operation while back propagation is performed, which Figure 4 (a) Compared with the traditional optimization strategy, the ΔT delay of each iteration is effectively reduced.

[0091] Step 4.2 iterative training

[0092] The iterative training process comprises: dividing the small batch data (B) into m micro-batch data, performing the corresponding part of the micro-batch data in each group, then calling the step 3.3 pipeline scheduling method for training, while calculating the loss function loss of each micro-batch data, performing reverse propagation calculation according to the loss function loss, synchronizing the gradient of the corresponding partition in the group, and based on the last task P4 of the first partition in step 4.1, after completion, the global synchronization update of the inter-group gradient is performed based on data parallelism. Finally, the model parameter is updated. The above process is an iterative calculation process, and after all the iterative calculations are completed, the final model parameters are output.

[0093] The application proposes a mixed parallel deep convolutional neural network parallel training method, which accelerates the large model training process of large data sets. The method has two components: intra-group parallelism and inter-group parallelism. An intra-group pipeline model parallel method is proposed to train large models and improve GPU utilization. First, an automatic model partitioning method is used to divide the model into multiple partitions, and the multiple partitions are placed on different GPUs. In a group, the intra-group pipeline model parallel method can effectively allow the pipeline training of large models. Inter-group parallelism provides a method to increase parallelism to accelerate the training of large DCNN models. This method is a partition-based overlapping computation strategy that partitions between groups to reduce communication time overhead. Compared with the traditional gradient synchronization strategy, the performance of the proposed overlapping computation and communication strategy is significantly improved (as shown in Figure 5 The performance of the mixed parallel deep convolutional neural network parallel training method proposed by the application is evaluated and tested by using the ImageNet standard data set on an 8-GPU server. The experimental results show that the performance of the method proposed by the application is better than that of the traditional data parallel (DP) and the most advanced mixed parallel training method torchgpipe. When the training reaches the same Top-1 accuracy, the acceleration performance of the method proposed by the application is improved by 59.6% and 14.3% (as shown in Figure 6 Therefore, the method has broad application prospects in deep learning parallelization.

[0094] Embodiment

[0095] The method proposed by the application is a general deep learning training acceleration method, which can be applied to many application scenarios. The embodiment details the process in the apple leaf disease detection model training process. (1) Data preprocessing, first, five kinds of apple leaf disease images collected manually (as shown in Figure 7The multi-dimensional matrix of the required data set is obtained by random cropping to 224x224 size, horizontal rotation, image conversion to a Tensor tensor, and normalization processing. (2) Model selection, ResNet-50 classification model is selected. (3) Model training, the training process requires grouping resource configuration, automatic model division, in-group pipeline model parallelism, and inter-group data parallelism based on calculation and communication overlap. The ResNet-50 classification model is trained on the apple leaf disease data set on a server with 8 GPUs (Tesla T4). First, the grouping resource configuration and the small batch data size need to be determined. The embodiment uses a 2-2 configuration (i.e., 2 groups, 2 GPU devices per group, a total of 4 GPUs), and the small batch data size is set to 128 (multi-dimensional matrix data generated by the disease image). Second, the model is divided. The ResNet-50 model includes 183 basic layers of convolution, pooling, activation, and full connection. The time of the 183 basic layers in the ResNet-50 model is counted by layer, and a time list with a length of 183 is generated. Each group has 2 GPU devices, and the output of the model automatic division method is [64, 119], indicating that the first 64 basic layers of the ResNet-50 network are placed in GPU0 in the group, and the last 119 basic layers are placed in GPU1 in the group. Then, the multi-dimensional matrix data fed into the training process is further divided, and the 128-size small batch data is divided into 12 micro batch data. According to the pipeline scheduling method, one micro batch data is fed into the pipeline every clock cycle, and forward propagation and reverse calculation are performed in a pipeline parallel manner. Each group is the same. Finally, in the process of inter-group data parallel back propagation, when the last micro batch data in the group is calculated in a certain partition, the hook function on the partition is triggered, and then all tensors of the same type on the partition are merged into a large tensor to perform inter-group all-reduce set communication asynchronously, while the original reverse calculation continues to be executed, greatly reducing the communication tensor fragmentation and improving the communication efficiency. (4) Disease identification, the disease classification model obtained based on the apple leaf disease data set on the ResNet-50 network using the training method proposed in the embodiment has an accuracy of 97.6%, reaching the standard level. The training time required without using the training method proposed in the embodiment is 80.04 hours, and the training time required using the training method proposed in the embodiment is 50.13 hours, with a performance improvement of 62.6%. Therefore, the method has a broad application prospect in deep learning model parallel training.

[0096] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A parallel training method for deep convolutional neural networks based on hybrid parallelism, characterized in that: The following steps are involved: Step 1: Input data and divide the data into several small batches; Step 2: The small batch data is divided into m micro batches, and each micro batch data is input into a group; Step 3: Perform forward calculation and reverse calculation on the micro-batch data in sequence; In each group, the GPU of the previous partition performs forward calculations on the previous micro-batch data and passes the calculation results to the GPU of the next partition. The GPU of the next partition continues forward calculations based on the calculation results of the GPU of the previous partition, and the GPU of the previous partition performs forward calculations on the next micro-batch data. After all GPUs in the group complete the forward calculation of each micro-batch, all GPUs perform reverse calculation. The GPU in the previous partition of each group performs reverse calculation on the previous micro-batch and passes the calculation result to the GPU in the next partition. The GPU in the next partition continues the reverse calculation based on the calculation result of the GPU in the previous partition. The GPU in the previous partition performs reverse calculation on the next micro-batch. Each partition GPU obtains the corresponding gradient through reverse calculation. Forward and backward computations are performed synchronously until all micro-batches of data in the mini-batch data corresponding to the group are calculated. K partitioned GPUs within a group perform synchronous computations. N groups perform synchronous computations on their respective mini-batches. A partitioned GPU corresponds to a zone in a deep convolutional neural network, and a partitioned GPU contains one GPU. A zone in a deep convolutional neural network contains several layers of the neural network. Step 4: After all GPUs in the same zone of different groups have completed their calculations, they synchronize their gradients. Step 5: After all GPU gradients in the same layer are synchronized, repeat steps 2 to 4 until all data calculations are completed.

2. The method for parallel training of a deep convolutional neural network based on hybrid parallelism according to claim 1, characterized in that: The process of determining k partition GPUs within a group is as follows: (1) Calculate the computational execution time of each layer of a deep convolutional neural network based on small batches of data; (2) Normalize the execution time of each layer; (3) There are k partitioned GPUs in a group, and several network layers of the deep convolutional neural network are randomly and evenly assigned to each of the k partitioned GPUs; (4) Obtain the running computing time of each partition, and adjust and replace the network layer in each partition through a double-layer round-robin method; (5) Taking the running calculation time of the partition as the benchmark, when the partition P with the largest sum of execution time max The partition P with the smallest sum of execution time min If the difference is less than a threshold, step (4) ends and the final k partitioned GPUs are obtained; (6) Input the results of k partitions into the GPU.

3. The method for parallel training of a deep convolutional neural network based on hybrid parallelism according to claim 2, characterized in that: The double-layer loop method includes an inner loop and an outer loop, wherein the outer loop is used to find the largest partition P. max , the inner loop is used to find the smallest partition P min .

4. The method for parallel training of a deep convolutional neural network based on hybrid parallelism according to claim 2, characterized in that: In step (5), the threshold is the maximum execution time in the model network layer.

5. The method for parallel training of a deep convolutional neural network based on hybrid parallelism according to claim 2, characterized in that: In step (2), the normalized calculation formula is: Among them, t i The execution time of each layer of the network; min(T) represents the network model layer with the shortest execution time; max(T) represents the network model layer with the longest execution time.

6. The method for parallel training of a deep convolutional neural network based on hybrid parallelism according to claim 1, characterized in that: In step 3, each partitioned GPU has a worker thread, and each worker thread includes a pair of task queues: in_queue i ,out_queue i ,i=1,2,...,k The in_queue i is the input queue, out_queue i For the output queue.

7. The method for parallel training of a deep convolutional neural network based on hybrid parallelism according to claim 6, characterized in that: In step 3, the process of transferring the micro-batch data calculation results of the previous partition GPU to the next partition GPU is as follows: The out_queue of the j-1 layer GPU j-1 F in i,j-1 The copy output is sent to the in_queue of the GPU in the jth layer j , put the GPU calculation results of layer j into out_queue j .

8. The method for parallel training of a deep convolutional neural network based on hybrid parallelism according to claim 1, characterized in that: In step 3, the loss function of each micro-batch data is calculated at the same time, and the back propagation is calculated based on the loss function.

9. The method for parallel training of a deep convolutional neural network based on hybrid parallelism according to claim 1, characterized in that: In step 4, a hook function is set for each partition, and the gradient synchronization is performed on GPUs in the same partition of different groups through the hook function.

Images