A hybrid pipeline parallel approach to accelerate distributed deep neural network training

By establishing a hierarchical cumulative distribution function model and dynamic programming algorithm in deep learning, optimizing model division and task placement, the problem of low GPU utilization in distributed deep learning is solved, and more efficient training speed is achieved.

CN112784968BActive Publication Date: 2024-12-03SOUTHEAST UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202110134151.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-01-29
Publication Date
2024-12-03
Estimated Expiration
2041-01-29

AI Technical Summary

Technical Problem

In the existing distributed deep learning training mode, the model division method is not complete enough, resulting in low GPU utilization and slow training speed. Especially when combining mixed parallelism and pipeline parallelism, there is a lack of effective algorithm implementation to improve training speed.

Method used

A hybrid model division and task placement method is proposed. By establishing a hierarchical cumulative distribution function model of deep neural networks, combining dynamic programming algorithms, model division and task placement are optimized, to ensure load balancing in each stage and improve GPU utilization.

Benefits of technology

The speed of distributed deep learning training is effectively improved. By combining mixed parallelism and pipeline parallelism, the GPU idle time is reduced and the training efficiency in the GPU cluster environment is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112784968B_ABST
    Figure CN112784968B_ABST
Patent Text Reader

Abstract

The present invention provides a hybrid pipeline parallel method for accelerating the distributed training of deep neural networks, which mainly solves the problem that resources are not fully utilized in the distributed training process of traditional GPU clusters and efficient distributed training cannot be achieved. The core mechanism of the present invention mainly includes three parts, namely deep learning model characterization, model hybrid partitioning and hybrid pipeline parallel partitioning. The present invention firstly targets the resource requirements of deep learning applications in the GPU training process, characterizes the corresponding indicators such as the amount of calculation, the number of intermediate result communications, and the amount of parameter synchronization during the training process, and uses them as inputs for model hybrid partitioning and task placement. Then, according to the model characterization results and the environment of the GPU cluster, two partitioning algorithms based on dynamic programming are designed to realize model hybrid partitioning and hybrid pipeline parallel partitioning, with the aim of minimizing the maximum value of the task execution time of each stage after the partition, so as to ensure load balancing and realize efficient distributed training of deep neural networks.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Field

[0002] The present invention relates to a model partitioning and task placement method in hybrid pipeline distributed deep learning, belonging to the technical field of distributed computing. Background Art

[0003] Deep learning is a general term for a class of pattern analysis methods. Through deep neural networks, multi-layer nonlinear information is used for supervised or unsupervised feature extraction and transformation. In recent years, with the continuous advancement of technology, deep learning has been widely used in many fields such as image recognition, natural language processing, and human-computer confrontation. However, with the continuous development of deep learning, the depth of the network is getting deeper and deeper, the number of layers is increasing, and the training time is also increasing. It usually takes dozens of hours or even weeks or months to train a complete deep neural network model. Therefore, how to efficiently execute the training process and reduce the training time has become a hot topic of research in recent years.

[0004] The training of deep neural networks mainly consists of three parts: (1) Forward propagation. Forward propagation refers to the process in which the neural network reads labeled data from the training set, operates through the hidden layer and activation layer, and finally outputs the result from the output layer. In order to speed up the training and reduce the impact of noise, one set of data is usually processed in one iteration. (2) Backward propagation. After the forward propagation outputs the result, it needs to be compared with the label of the training set itself, and the error between the training value and the true value is calculated using a suitable function. Since the weight of each parameter on the error is different, in order to calculate the weight, the gradient of the parameters of each layer is calculated in turn from the reverse direction of the forward propagation using the chain rule. The intermediate result of the forward propagation of the corresponding layer is needed to calculate the gradient. (3) Gradient descent. According to the error between the calculated value and the true value and the gradient of the parameters of each layer solved in the back propagation, the parameters of each layer are gradient descended, the parameters are updated, and one iteration is completed. Through the continuous iteration of these three steps, the neural network finally reaches a converged state.

[0005] As models continue to develop, the number of model layers increases, and the required computing power and video memory usage increase, distributed machine learning has become a key issue of concern. Traditional distributed machine learning can be divided into two main methods: (1) Data parallelism. Data parallelism means that on multiple GPUs, each GPU contains a complete copy of the model, using partial data and training relatively independently. After an iteration is completed, the parameter server collects the parameters on all GPUs and then performs parameter synchronization updates. As GPU computing power continues to improve, computing time continues to be shortened. At this time, the communication time required for parameter synchronization becomes a bottleneck in data parallelism, greatly reducing the performance of distributed training. (2) Model parallelism. Model parallelism splits the model from the middle and divides it into different GPUs. Each GPU only has a part of the complete model. During forward propagation, the first part of the model, stage 1, is calculated first, and then the intermediate result is sent to the second part, stage 2, to continue calculation. The same is true for back propagation and gradient descent. However, since the forward propagation of a stage needs to wait for the result of the back propagation of the subsequent stage, the GPU is idle during the waiting time, which greatly reduces the performance of distributed computing. At the same time, how to reasonably divide the model so that each stage achieves load balancing is also a difficulty that users need to consider.

[0006] In view of the characteristics of these two parallel methods, researchers have recently proposed two ways to improve distributed machine learning. One is hybrid parallelism. Traditional data parallelism is divided from the batch dimension of data, while hybrid parallelism is divided from the perspective of tensors, which expands the parallel method, thereby reducing communication bottlenecks and improving GPU utilization. The other is pipeline parallelism, which is an improvement on model parallelism. In the forward propagation process of model parallelism, after a GPU completes the calculation of a set of data, it no longer waits for backpropagation, but continues to calculate the forward propagation process of the next set of data. By reasonably allocating the execution order of forward and backpropagation, theoretical load balancing is achieved, which greatly reduces the idle time of the GPU and improves parallel efficiency. However, in existing work, these two parallel methods have not been comprehensively considered, resulting in incomplete strategy division and insufficient GPU utilization. Therefore, how to combine hybrid parallelism and pipeline parallelism to design an effective algorithm to achieve model division and thus improve training speed has become a key issue. Summary of the invention

[0007] The present invention mainly aims at accelerating the training of distributed deep learning under the current pipeline training mode, and proposes a hybrid model partitioning and task placement method. In the hybrid pipeline mode, in the same time slice, the GPU performs corresponding calculations of different parts of the model in different stages. In order to make the model training highly parallel and improve the speed of distributed training, it is necessary to consider not only the training speed but also the load balancing between stages and the hybrid parallel mode when dividing the model. To this end, this method first solves the result of hybrid parallelism based on the hierarchical characteristics of the deep neural network, and then uses dynamic programming to solve the load-balanced model partitioning and task placement based on this result and the characteristics of pipeline parallelism.

[0008] In order to achieve the above object, the technical solution of the present invention is as follows: a hybrid pipeline parallel method for accelerating distributed deep neural network training, the method comprising the following steps:

[0009] Step 1: Establish a hierarchical cumulative distribution function (CDF) model of the deep neural network, analyze the corresponding input conditions required for the deep learning application execution model partitioning and task placement algorithm, and use the pytorch framework to obtain the parameter quantity of each layer; then calculate the intermediate result communication volume of each layer according to the given batch-size; finally, calculate the floating-point calculation amount of each layer according to the type of each layer of the network, such as convolutional layer, fully connected layer, etc., to prepare for step 2;

[0010] Step 2: Based on the result of step 1, use the dynamic programming algorithm to solve the parallel time when different hybrid parallel methods are used between any two layers of the deep neural network, in preparation for step 3;

[0011] Step 3: Based on the result of the hybrid parallel mode solved in step 2, a dynamic programming algorithm is used to divide the process into different stages. Each stage contains some continuous layers of the neural network, and a hybrid parallel method is given in the stage. The purpose is to minimize the maximum value of the task execution time of each stage after the division, so that the GPU load is as balanced as possible in the hybrid parallel mode, and the GPU utilization rate is improved to accelerate distributed training.

[0012] Wherein, the step 1 is specifically as follows:

[0013] A cumulative distribution function model is established for the neural network hierarchical model. Under the premise of given hyperparameters, the corresponding indicators of the neural network hierarchical structure are characterized according to the device topology D:

[0014] a describes the total forward and reverse calculation time T of each layer of the neural network cp ;

[0015] b describes the input size of each layer of the neural network i ;

[0016] c describes the output of each layer of the neural network Sizeout i ;

[0017] d describes the parameters w of each layer of the neural network i .

[0018] Wherein, the step 2 is specifically as follows:

[0019] Hybrid parallel partitioning algorithm, the goal of hybrid parallel partitioning is to solve the optimal hybrid parallel mode between any two layers. Under the premise of a given bandwidth B, according to the idea of ​​comprehensive dynamic programming, this problem is converted into finding the parallel time of hybrid parallelism between any one layer to the last layer. By retaining the intermediate results, the optimal hybrid parallel strategy between any two layers can be found by backtracking. According to Sizein i , Sizeout i , w i Calculate the communication volume required for different divisions of each layer, calculate the communication time according to the bandwidth B, and select the optimal strategy for the current layer. A(i,j,m) represents the optimal strategy for mixed parallelism of layer i→j using m cards, and T(k,m,TYPE) represents the time for mixed parallelism when layer k uses m cards and the parallel mode is TYPE.

[0020]

[0021] In step 3, based on the hybrid parallel partitioning result of step 2, the model is divided into multiple stages using a dynamic programming algorithm. Each stage is a part of the continuous layer in the neural network. The optimal hybrid parallel method is used within the stage, and the pipeline model is used between the stages. i , Sizeout i , w i , B calculates the communication time of each partition. The partitioning goal is to minimize the maximum value of the calculation and communication time of each stage after the partition, improve the GPU utilization rate, speed up the training speed, and let Cp s is the computation time of stage s using optimal hybrid parallelism, Cm s The communication time of the optimal hybrid parallelism is adopted for stage s. In pipeline mode, the total time can be converted into the time taken by the stage with the largest sum of computation and communication time among all stages;

[0022]

[0023] Through the dynamic programming method, the time used by the stage with the largest sum of calculation and communication time in all stages is solved, so that the time of each stage is roughly equal. In the hybrid parallel mode, the GPU load is as balanced as possible.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows: the scheme provides a hybrid pipeline parallel method for accelerating the training of distributed deep neural networks, which can solve the problem that the traditional model division and task placement do not consider the model division method comprehensively, resulting in uneven load in the distributed deep learning process and insufficient GPU utilization. From the perspective of tensor calculation, a hybrid parallel method is proposed by dividing the tensor into different dimensions, which expands the traditional parallel method and increases the possibility of parallelism. Then, considering a variety of division methods, the model is divided into multiple stages, hybrid parallelism is adopted within the stage, and pipeline parallelism is adopted between the stages, so as to minimize the time used by the stage with the largest sum of calculation and communication time among all stages, thereby improving the training speed of pipeline distributed deep learning in the GPU cluster environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a schematic diagram of a GPU cluster of the present invention;

[0026] Figure 2 A schematic diagram of the hybrid parallel method of the present invention;

[0027] Figure 3 It is the overall flow chart of the present invention;

[0028] Figure 4 It is a schematic diagram of the overall design of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0030] Example 1: See Figure 1 , a hybrid pipeline parallel method for accelerating distributed deep neural network training, the method comprising the following steps:

[0031] Step 1: Establish a hierarchical cumulative distribution function (CDF) model of the deep neural network, analyze the corresponding input conditions required for the deep learning application execution model partitioning and task placement algorithm, and use the pytorch framework to obtain the parameter quantity of each layer; then calculate the intermediate result communication volume of each layer according to the given batch-size; finally, calculate the floating-point calculation amount of each layer according to the type of each layer of the network, such as convolutional layer, fully connected layer, etc., to prepare for step 2;

[0032] Step 2: Based on the result of step 1, use the dynamic programming algorithm to solve the parallel time when different hybrid parallel methods are used between any two layers of the deep neural network, in preparation for step 3;

[0033] Step 3: Based on the result of the hybrid parallel mode solved in step 2, a dynamic programming algorithm is used to divide the training into different stages. Each stage contains some continuous layers of the neural network, and a hybrid parallel method is given in the stage. The purpose is to minimize the maximum value of the task execution time of each stage after the division, so that the GPU load is as balanced as possible in the hybrid parallel mode, and the GPU utilization rate is improved to accelerate distributed training.

[0034] Wherein, the step 1 is specifically as follows:

[0035] A cumulative distribution function model is established for the neural network hierarchical model. Under the premise of given hyperparameters, the corresponding indicators of the neural network hierarchical structure are characterized according to the device topology D:

[0036] a describes the total forward and reverse calculation time T of each layer of the neural network cp ;

[0037] b describes the input size of each layer of the neural network i ;

[0038] c describes the output of each layer of the neural network Sizeout i ;

[0039] d describes the parameters w of each layer of the neural network i .

[0040] Wherein, the step 2 is specifically as follows:

[0041] Hybrid parallel partitioning algorithm, the goal of hybrid parallel partitioning is to solve the optimal hybrid parallel mode between any two layers. Under the premise of a given bandwidth B, according to the idea of ​​comprehensive dynamic programming, this problem is converted into finding the parallel time of hybrid parallelism between any one layer to the last layer. By retaining the intermediate results, the optimal hybrid parallel strategy between any two layers can be found by backtracking. According to Sizein i , Sizeout i , w i Calculate the communication volume required for different divisions of each layer, calculate the communication time according to the bandwidth B, and select the optimal strategy for the current layer. A(i,j,m) represents the optimal strategy for mixed parallelism of layer i→j using m cards, and T(k,m,TYPE) represents the time for mixed parallelism when layer k uses m cards and the parallel mode is TYPE.

[0042]

[0043] In step 3, based on the hybrid parallel partitioning result of step 2, the model is divided into multiple stages using a dynamic programming algorithm. Each stage is a part of the continuous layer in the neural network. The optimal hybrid parallel method is used within the stage, and the pipeline model is used between the stages. i , Sizeout i , w i , B calculates the communication time of each partition. The partitioning goal is to minimize the maximum value of the calculation and communication time of each stage after the partition, improve the GPU utilization rate, speed up the training speed, and let Cp s is the computation time of stage s using optimal hybrid parallelism, Cm s The communication time of the optimal hybrid parallelism is adopted for stage s. In pipeline mode, the total time can be converted into the time taken by the stage with the largest sum of computation and communication time among all stages;

[0044]

[0045] Through the dynamic programming method, the time used by the stage with the largest sum of calculation and communication time in all stages is solved, so that the time of each stage is roughly equal. In the hybrid parallel mode, the GPU load is as balanced as possible.

[0046] Specific embodiment: The present invention is mainly carried out in a GPU cluster environment.

[0047] Figure 1 It shows a schematic diagram of a GPU cluster, which mainly includes several GPU server nodes. Each server node has several GPUs. The nodes are connected internally through PCIe, and the nodes are connected through Ethernet and other means.

[0048] Figure 2 It shows a schematic diagram of a hybrid parallel method. The same rectangle is divided into two parts, representing the division method. The rectangles of different parts stacked together represent the repeated calculation part, that is, the part that needs communication. The three figures correspond to three tensor division methods, which require communication weight parameters, output layer data, and input layer data respectively.

[0049] Figure 3 It shows the overall flow chart. First, the system is initialized, input data is loaded and the specified model is created. Then the model is characterized by layers and the cumulative distribution function model is calculated. Then hybrid parallel partitioning is performed according to the GPU cluster environment. Finally, the hybrid parallel partitioning result is used to design and partition the various stages of pipeline parallelism. Each stage contains some continuous layers of the neural network, and the hybrid parallel method used in the stage is given. The purpose is to minimize the maximum value between the execution time of tasks in each stage after the partition and give the final strategy.

[0050] In the model characterization stage, in terms of the amount of calculation, the torchstat library is used to print out the amount of calculation required for the matrix calculations (mkldnn_convolution, max_pool2d_with_indices, batch_norm, addmm, etc.) of the neural network to perform forward propagation and reverse propagation on the GPU in sequence, and then the calculation time is obtained according to the GPU computing power. For the communication volume, the matrix shape of each layer output and the number of parameters of each layer of the neural network are also printed out, and then the intermediate results and parameter sizes of each layer are calculated according to the bytes required for floating-point numbers. After pre-training, it is obtained: the total forward and reverse calculation time T of each layer of the neural network characterization cp ; Sizein of intermediate results of each layer of neural network i , Sizeout i ; Neural network parameters w i .

[0051] In the hybrid parallel partitioning stage, the goal is to solve the optimal hybrid parallel method between any two layers. According to the idea of ​​dynamic programming, this problem is converted into finding the parallel time of hybrid parallelism between any layer to the last layer. By retaining the intermediate results, the optimal hybrid parallel strategy between any two layers can be found by backtracking.

[0052] In the hybrid pipeline parallel partitioning stage, the model is divided into multiple stages, each stage is a part of the continuous layer in the neural network. The optimal hybrid parallel method is used within the stage, and the pipeline model parallelism is used between the stages. The partitioning goal is to minimize the maximum value of the calculation and communication time of each stage after the partition, improve the GPU utilization, and speed up the training speed.

[0053] For the hybrid parallel partitioning and hybrid pipeline parallel partitioning problems, a mathematical model is established:

[0054] 1) Physical resource modeling in GPU cluster:

[0055] variable meaning D GPU cluster equipment B GPU cluster bandwidth

[0056] Table 1 Physical resource modeling results

[0057] 2) Neural network modeling:

[0058]

[0059]

[0060] Table 2. Results of neural network computing demand modeling.

[0061] a. Traffic volume of layer i:

[0062]

[0063] b. Total time of layer i

[0064]

[0065]

[0066] T=T cm +T cp

[0067] c. For table i→j, the optimal strategy is to use m cards for mixed parallelism:

[0068]

[0069] d. The overall goal is to minimize the total time of an iteration, which can be converted into the time taken by the stage with the largest sum of the computation and communication time in all stages.

[0070]

[0071] The pseudo code for the hybrid parallel partitioning phase is as follows:

[0072]

[0073]

[0074] For Algorithm 1, the input is the parameters of each layer of the neural network Layer, the number of GPUs M, and the GPU cluster bandwidth M. The algorithm output is the optimal hybrid parallel method between any two layers. The first and second lines of Algorithm 1 use a for loop to traverse the hybrid parallel situation from any layer to the last layer under any device. Since this is a sub-optimal problem, the idea of ​​dynamic programming can be used to simultaneously derive the hybrid parallel situation between any two layers under any device. Lines 4 to 6 specifically calculate the time required for hybrid parallelism. Select a parallel method at the current layer, according to Sizein i , Sizeout i , w i Calculate the communication volume required for different divisions of each layer, and calculate the total time T required for the three parallel methods of the previous layer according to the bandwidth B cp +T cm , take the minimum value. Then traverse all the parallel methods of the current layer, save the one with the shortest time, and then record it. After the loop is completed, output the optimal hybrid parallel method between any two layers.

[0075] Algorithm 2 Hybrid pipeline parallel partitioning Input: The result of hybrid parallelism in Algorithm 1, the number of GPUs M, and the bandwidth of the GPU cluster M Output: Hybrid pipeline parallel partitioning result 1: functionCOMPUTE PARTITIONING(Hybrid; M) 2:end=len(Hybrid) 3: fori=0toend; j=itoend; m=1toMdo 4:Initialize Split(i;j;m)=(INF;0;0) 5:fork=itoj;n=1tom do 6:New Split(i;j;m)=Hybrid(i;j;k;n) 7:ifNewSplit.time<Split.timethen 8:Split(i;j;m)=(time;k;n) 9:endif 10:endfor 11:endfor 12: k = 0; m = M 13:whilek<end and m<M do 14:Split layer[k;end]from Split(k;end;m):split 15:m=m-Split(k;end;m):machine used 16:k=Split(k;end;m):split 17:endwhile 18:endfunction

[0076] The input of Algorithm 2 is the result of hybrid parallelism in Algorithm 1, the number of GPUs M, and the bandwidth of the GPU cluster M. The output is the result of hybrid pipeline parallel partitioning. Lines 2 to 5 of Algorithm 2 initialize the parameters, use a for loop, loop through any i to j layers, and split from the middle k layer. Layers i to k use m cards for hybrid parallelism. Lines 6 to 8, after each partition, according to Sizein i , Sizeout i , w i Calculate the amount of communication required for the partition, and then calculate the time T required for parallelization based on the bandwidth B cm , if it is better than the previous time, update the parallel mode and store it in the table. From line 12 to line 17, perform hybrid pipeline parallel partitioning. First, from the i~end layer, according to the table established above, select the best partitioning method and use m block cards to split from the k layer. Then, for the k~end layer, Mm block cards, use a similar method to partition until the partition is completed. Finally, output the hybrid pipeline parallel partitioning result.

[0077] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention. These corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A hybrid pipeline parallel method for accelerating distributed training of deep neural networks, characterized by: The method comprises the following steps: Step 1: Establish a hierarchical cumulative distribution function (CDF) model of the deep neural network, analyze the corresponding input conditions required for the deep learning application execution model partitioning and task placement algorithm, and use the pytorch framework to obtain the parameter quantity of each layer; then calculate the intermediate result communication volume of each layer according to the given batch-size; finally, calculate the floating-point calculation amount of each layer according to the type of each layer of the network, in preparation for step 2; Step 2: Based on the result of step 1, use the dynamic programming algorithm to solve the parallel time when different hybrid parallel methods are used between any two layers of the deep neural network, in preparation for step 3; Step 3: Based on the result of the hybrid parallel mode solved in step 2, a dynamic programming algorithm is used to divide the training into different stages. Each stage contains some continuous layers of the neural network, and the hybrid parallel mode used in the stage is given. The purpose is to minimize the maximum value of the task execution time of each stage after the division, so that the GPU load is as balanced as possible in the hybrid parallel mode, and the GPU utilization rate is improved to accelerate distributed training. The step 1 is specifically as follows: A cumulative distribution function model is established for the neural network hierarchical model. Under the premise of given hyperparameters, the corresponding indicators of the neural network hierarchical structure are characterized according to the device topology D: a. Characterize the total forward and reverse calculation time T of each layer of the neural network cp ; b. Characterize the input size of each layer of the neural network i ; c. Characterize the output size of each layer of the neural network i ; d. Characterize the parameters w of each layer of the neural network i ; The step 2 is specifically as follows: Hybrid parallel partitioning algorithm, the goal of hybrid parallel partitioning is to solve the optimal hybrid parallel mode between any two layers. Under the premise of a given bandwidth B, according to the idea of ​​comprehensive dynamic programming, the problem is converted into finding the parallel time of hybrid parallelism between any one layer to the last layer. By retaining the intermediate results, the optimal hybrid parallel strategy between any two layers can be found by backtracking. According to Sizein i , Sizeout i , w i Calculate the communication volume required for different divisions of each layer, calculate the communication time according to the bandwidth B, and select the optimal strategy for the current layer. A(i,j,m) represents the optimal strategy for mixed parallelism of layer i→j using m cards, and T(k,m,TYPE) represents the time for mixed parallelism when layer k uses m cards and the parallel mode is TYPE. In the step 3, based on the hybrid parallel partitioning result in step 2, the model is divided into multiple stages using a dynamic programming algorithm. Each stage is a part of the continuous layer in the neural network. The optimal hybrid parallel method is used within the stage, and the pipeline model is used in parallel between stages. i , Sizeout i , w i , B calculates the communication time of each partition. The partitioning goal is to minimize the maximum value of the calculation and communication time of each stage after the partition, improve the GPU utilization rate, speed up the training speed, and let Cp s is the computation time of stage s using optimal hybrid parallelism, Cm s The communication time of the optimal hybrid parallelism is adopted for stage s. In pipeline mode, the total time can be converted into the time taken by the stage with the largest sum of computation and communication time among all stages; Through the dynamic programming method, the time used by the stage with the largest sum of calculation and communication time in all stages is solved, so that the time of each stage is roughly equal. In the hybrid parallel mode, the GPU load is as balanced as possible.

Citation Information

Patent Citations

  • GPU memory optimization method and system for a deep learning training task

    CN109919310A

  • Heterogeneous network perception model division and task placement method in pipelined distributed deep learning

    CN110533183A