Pipeline Parallel Methods for Accelerating Neural Network Training in Heterogeneous GPU Clusters

Abstract

This invention provides a pipelined parallel method for accelerating neural network training in heterogeneous GPU clusters. The core mechanism mainly comprises three parts: deep learning model characterization, homogeneous GPU model partitioning and task placement, and heterogeneous GPU model partitioning and task placement. First, this invention addresses the resource requirements of deep learning applications during GPU training, characterizing corresponding indicators such as computational load, intermediate result communication volume, and parameter synchronization volume when using different types of GPUs, and using these as inputs for model partitioning and task placement. Then, based on the model characterization results and the GPU cluster environment, a dynamic programming-based partitioning algorithm is designed to achieve heterogeneous GPU and heterogeneous bandwidth-aware pipelined hybrid parallel model partitioning and task placement. The aim is to minimize the maximum execution time of each stage of the task after partitioning, ensuring load balancing and achieving efficient distributed training of deep neural networks.

Description

Technical Field

[0001] This invention relates to a pipelined parallel method for accelerating neural network training in heterogeneous GPU clusters, belonging to the field of distributed computing technology. Background Technology

[0002] In recent years, artificial intelligence has developed rapidly, and deep neural networks (DNNs), as the most commonly used technology in AI, have been widely applied in image processing, natural language processing, and autonomous driving. Deep neural networks are a technique that uses artificial neural networks as a basic architecture to learn representations from training data. Its training process mainly consists of the following three parts:

[0003] (1) Forward propagation. Forward propagation refers to the process by which a neural network reads data from a labeled training set, performs calculations through intermediate layers, and finally outputs the calculation results from the output layer. In order to reduce noise during training and improve training speed, each forward propagation processes a batch of data.

[0004] (2) Backpropagation. After the forward propagation process is completed, the result of the forward propagation is compared with the label on the training set, and the error is calculated using the corresponding loss function. Then, the gradient of each parameter is calculated from the reverse direction of the forward propagation using the chain rule, and the weight of the influence of different parameters on the error is calculated. The intermediate results of the forward propagation of the corresponding layer are used during backpropagation.

[0005] (3) Gradient Descent. Based on the parameter gradients obtained in backpropagation, gradient descent is used to update the parameters of each layer, completing one training iteration. This iterative process is repeated until the neural network reaches convergence, completing the training. However, with the continuous development of deep learning, in order to obtain higher training accuracy, the number of layers in deep neural networks is constantly increasing, gradually exposing the problem of insufficient computing resources during training. Deep learning training requires powerful computing support. In recent years, new models have emerged continuously, and the amount of computation required for training has also increased explosively. The increase in computation has led to a surge in the time required for training, making traditional single-card training infeasible. How to use GPU clusters for distributed training has become a key issue of concern.

[0006] Traditional distributed training methods mainly fall into two categories:

[0007] (1) Data parallelism refers to the process in which each GPU computing node has a complete copy of the neural network model during training. During each training iteration, different nodes independently use a portion of the data to perform forward and backward propagation processes and calculate the parameter gradients. After one iteration, all nodes synchronize the gradients they have calculated and then continue to execute the next iteration. However, since each node has a complete copy of the model, data parallelism cannot reduce the use of GPU memory. Moreover, as the degree of parallelism increases, the amount of parameter communication gradually increases, and the limited communication bandwidth will cause the parameter synchronization time to become the bottleneck of the entire training, affecting the training efficiency.

[0008] (2) Model Parallelism. Model parallelism divides the complete model into multiple parts and distributes them to different GPU computing nodes. During forward propagation, the intermediate results generated by the upstream node after computation are transmitted to the downstream node, while during backward propagation, the gradients generated by each node are propagated in the opposite direction. Because model parallelism divides the network, each GPU is only responsible for computing a part of the model, which can reduce the use of GPU memory. During training, forward and backward propagation are performed according to the order of the neural network's computation graph. After each node completes its computation, it transmits the computation result to the downstream node until a round of data iteration is completed. However, due to the computational dependency, backward propagation can only be performed after the forward propagation process is completed, resulting in a large amount of idle time on the GPU and very low distributed training efficiency.

[0009] To address the characteristics of these two parallel approaches, researchers have recently proposed pipelining hybrid parallelism. Pipelining hybrid parallelism is a popular training method that combines traditional data parallelism and model parallelism. It further divides the data, with each part processed sequentially by a portion of the model on different training nodes in a pipeline-like manner, improving equipment utilization and training speed. In pipelining parallelism, the divided model portions are called stages; data is processed by each stage and then passed to the next. However, pipelining parallelism requires that each stage derived from the model division has as similar a load as possible, meaning that the execution time of each stage should be as similar as possible. Otherwise, it will lead to a decrease in equipment utilization and model training speed. Therefore, a suitable model partitioning and task placement method is needed, comprehensively considering the partitioning of neural network model layers and the corresponding equipment selection for training tasks to balance the load of each stage and ensure high training efficiency. Thus, pipelining hybrid parallel training achieves load balancing by employing data parallelism in the divided stages to accelerate the training speed of individual stages. Furthermore, existing research has also considered the potential heterogeneity of network bandwidth in the cluster and implemented automatic, load-balanced model partitioning and task placement through a dynamic programming-based approach. Under load balancing conditions, the pipeline can ensure high equipment utilization, thereby improving training speed.

[0010] Due to the rapid iteration of GPU products, cost-efficiency issues, and the combined consideration of different acceleration requirements, the use of heterogeneous GPUs in real-world GPU clusters is unavoidable. In heterogeneous GPU clusters, there is simultaneous heterogeneity in both GPUs and network bandwidth. This means that different types of GPUs require different computation times to process the same layer of the same neural network model, and the communication time between stages, determined by bandwidth, also varies. However, existing model partitioning methods for heterogeneous environments only address the problem caused by bandwidth heterogeneity, failing to consider the pervasive GPU heterogeneity. Their model partitioning and task placement methods are designed based on the assumption that all GPUs in the cluster are homogeneous. This leads to the inability of existing methods to guarantee load balancing in pipeline stages within heterogeneous GPU clusters, resulting in reduced equipment utilization and decreased training speed. As shown in the figure below, under pipeline load imbalance caused by heterogeneous GPUs, the time required to complete each computation differs between different stages, thus reducing GPU utilization compared to a load-balanced state. Summary of the Invention

[0011] To address the problems existing in current technologies for pipelined parallel acceleration training in heterogeneous GPU cluster environments, and to ensure load balancing across pipeline stages while comprehensively considering GPU heterogeneity and bandwidth heterogeneity, this invention proposes a novel heterogeneous GPU-aware distributed pipeline training acceleration method. In hybrid pipeline mode, within the same time slice, GPUs execute different parts of the model's computations at different stages. To achieve high parallelism in model training and improve distributed training speed, model partitioning must consider not only training speed but also load balancing between stages. Therefore, this method, based on the hierarchical characteristics of deep neural networks, the hierarchical structure of the network topology in heterogeneous GPU clusters, and the homogeneity of GPUs within nodes while heterogeneity exists between nodes, first uses dynamic programming to solve for the partitioning result of using homogeneous GPUs for training within a node. Then, based on this result, dynamic programming is used again to solve for the model partitioning and task placement that achieves load balancing between nodes using heterogeneous GPUs and heterogeneous bandwidth.

[0012] To achieve the above objectives, the present invention provides the following technical solution:

[0013] A pipelined parallel method for accelerating neural network training in heterogeneous GPU clusters includes the following steps:

[0014] Part 1: Establishing a cumulative distribution function (CDF) model for the neural network hierarchy model. Given hyperparameters, analyzing the input conditions required for deep learning applications to perform model partitioning and task placement algorithms, analyzing the model, and characterizing the corresponding indicators of the neural network hierarchy structure based on the device topology, including the total forward and backward computation time, input quantity, output quantity, and synchronization parameter quantity of each layer of the neural network. Characterizing physical resources and neural network models for different types of GPUs; preparing for the next part.

[0015] Specifically, since neural network training involves forward propagation to predict results and backpropagation to update parameters, and data is computed layer by layer through the neural network, a cumulative distribution function model is established for the hierarchical structure of the neural network in pipeline mode. Based on the cumulative distribution function model and the floating-point computing power of different types of GPUs, the theoretical total computation time, activation value size, and number of parameters for each layer of the neural network during training are characterized when using each type of GPU.

[0016] Part Two: Solving for Homogeneous GPU Partitioning within Nodes. This part uses the metrics described in Part One as input. For each type of GPU, a dynamic programming algorithm is used to partition the pipeline into stages. Each stage contains consecutive layers of the neural network. Only one type of GPU is used at a time, and the decision on whether to use data parallelism within a stage is given. The goal is to minimize the maximum value between the task execution times of each stage after partitioning. The training time for training any number of consecutive layers of a neural network using this type of GPU is calculated. Finally, the optimal model partitioning and task placement scheme for training any number of consecutive layers of a deep neural network using each type of GPU is obtained, thus providing a foundation for subsequent pipeline stage partitioning considering GPU heterogeneity and network bandwidth heterogeneity between nodes.

[0017] In the problem of partitioning heterogeneous GPU cluster models, the partitioned stages are assigned to a GPU set, which may contain one or more GPUs, and the GPUs exhibit heterogeneity. Furthermore, the network connections within and between different cluster nodes are also heterogeneous. Therefore, it is impossible to map GPUs to a specific GPUID, transforming this problem into an NP-hard problem. To solve this NP-hard problem, this invention employs a layered solution based on the network topology partitioning during modeling. For the first layer topology... That is, within a node, the GPUs are homogeneous, so there is no need to consider the impact of heterogeneous GPUs. To use a GPU with m blocks of type t The minimum time required for pipelined parallel computation across layers is an NP-hard problem that cannot be solved directly. However, considering the hierarchical characteristics of neural networks and the operational features during training, we can decompose the entire problem into several subproblems and the data transfer problem between adjacent subproblems. This problem possesses the optimal substructure property: if the model is divided into n stages, this division is optimal, then any k (0 < k < n) stages forming a subproblem is also optimal. Therefore, we can try to solve this problem using dynamic programming. For computational convenience, we decompose the problem into only one subproblem at a time, namely the computation time of the data parallel stage. The recursive formula can be expressed by the following formula:

[0018]

[0019] in Representing the The layer uses a type of The optimal time subproblem is solved in parallel using GPUs. Represents data communication time. The representative will The time required for data parallelism of a multilayer neural network using m GPUs of type t. The type used in this partitioning is The number of GPUs, The type used in this partitioning is The remaining number of GPUs. Based on this formula, iterate through... The first-level topology can be solved by performing a dynamic programming search. This refers to the optimal solution for partitioning within a node, which prepares the groundwork for the next part.

[0020] Part 3: Model Partitioning and Task Placement Algorithm Based on Heterogeneous GPUs and Heterogeneous Bandwidth Awareness. The results from Part 2 are used for partitioning on homogeneous GPUs within a node. A dynamic programming algorithm is employed to partition the pipeline stages between nodes using heterogeneous GPUs and network bandwidth, based on the intra-node partitioning. The aim is to minimize the maximum difference in task execution time between stages after partitioning, ensuring load balancing between stages and improving GPU utilization to accelerate distributed training. In this step, firstly, the results from Part 2 are reviewed to obtain the optimal model partitioning and task placement scheme within the node for each type of GPU. Secondly, based on the intra-node partitioning results, a dynamic programming method is used to classify and consider the GPU types used in this partitioning, comparing the training time of the new partition stage, the bandwidth-based communication time at this partition point, and the maximum time required for the remaining stages, thus obtaining the shortest training time when heterogeneous GPUs and heterogeneous bandwidths exist. Finally, the dynamic programming results from step 3-2 are reviewed to obtain the optimal partition point, thus yielding the optimal model partitioning and task placement scheme.

[0021] Specifically, after solving the first-level topology in the second part, this part still needs to solve the second-level topology. Solving for . Definition for Layers, using The minimum training time for a single iteration after partitioning the GPU ensemble into a model. Following the previously proposed pipelined approach, the model is divided into multiple stages, each executed by a single GPU or multiple GPUs in data parallelism, with model parallelism between different stages. Due to GPU heterogeneity and the uncertainty of each stage, this problem cannot be directly solved. However, it can be noted that in the data parallelism stage, due to the different computing capabilities of heterogeneous GPUs, following the data parallel training mode, parameter synchronization will inevitably lead to a situation where the more powerful GPU finishes its computation, waiting for the less powerful GPU to finish before parameter synchronization can occur, resulting in a lagging phenomenon, reducing GPU utilization, and affecting the overall computation time. Therefore, this situation can be directly excluded in the algorithm partitioning to reduce the overall complexity; that is, only homogeneous GPUs can be used in the data parallelism stage. After pruning, similar to... This problem still possesses the substructure optimality property and can be solved using dynamic programming. However, due to the heterogeneity of GPUs, it is necessary to classify and discuss heterogeneous GPUs separately. The recursive formula is as follows:

[0022]

[0023] The right side of the equals sign has several possible cases, each representing a different type of GPU divided into data-parallel subproblems. Indicate arrive Layers The types are The time required for data parallelism to be performed on the nodes, traversing You can get The optimal solution for partitioning. After obtaining the optimal solution, it is necessary to backtrack the results to obtain the task placement scheme. The specific method is to first... The partitioning results are backtracked to obtain the optimal partitioning point and the GPU for partitioning, i.e., the value in the formula. and Continue backtracking on the subproblems until... That is, when no partitioning is required, we get The task placement scheme. Next, according to The task placement scheme will continue to be optimized. By backtracking, the final task placement result can be obtained. After model partitioning and task placement, the system will allocate tasks to the corresponding GPUs based on the model partitioning results, generate the model code required for execution at each stage, and perform pipelined parallel computing. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of a prototype system for a pipelined hybrid parallel method to accelerate distributed training in a heterogeneous GPU cluster.

[0025] Figure 2 This is a schematic diagram of a heterogeneous GPU cluster with two types of GPUs and three types of network bandwidth.

[0026] Figure 3 This is a schematic diagram of pipeline execution in load-balanced and load-unbalanced states.

[0027] Figure 4 This is a flowchart of the pipelined hybrid parallel method for accelerating distributed training in heterogeneous GPU clusters. Detailed Implementation

[0028] The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0029] The overall prototype structure of the system is as follows Figure 1 As shown, the entire system is logically divided into four layers. The physical resource layer mainly includes GPU computing resources, network resources, and storage resources involved in model training. The second layer is the basic framework layer, which mainly consists of existing technologies used in the system development process, including the deep learning training framework PyTorch, and libraries such as BLAS, CUDA, and cuDNN required for training deep neural networks using GPUs. Modules in the policy optimization layer are developed and implemented based on these technologies. The policy optimization layer then provides related key technical services, mainly including a heterogeneous GPU-aware model partitioning and task placement mechanism module, and a pipelined distributed training framework. These modules are based on the method proposed in this invention, namely, a pipelined hybrid parallel method for accelerating distributed training in heterogeneous GPU clusters. Finally, the application layer mainly includes a series of deep learning applications supported by the system, such as completing image recognition tasks by training image recognition models, such as the VGG16 model.

[0030] Figure 2This diagram illustrates a heterogeneous GPU cluster to which the present invention applies, exhibiting both GPU heterogeneity and network heterogeneity. In a heterogeneous GPU cluster, due to differences in computing power among GPUs, different types of GPUs require different training times to complete the training of the same layer of a neural network model. This means that model partitioning and task placement schemes based on homogeneous GPUs cannot guarantee that the training time for each stage is as similar as possible, i.e., they cannot guarantee load balancing across pipeline stages. Consequently, GPU resource utilization is low, which in turn affects the training speed of the neural network model. Figure 3 This diagram illustrates the execution of a pipeline under load-balanced and load-unbalanced conditions. It can be seen that the idle time of devices in the load-unbalanced state is much longer than that in the load-balanced state. This illustrates the importance of ensuring pipeline load balancing and the motivation for proposing this invention.

[0031] The overall execution flow of the pipelined hybrid parallel method for accelerating distributed deep neural network training in heterogeneous GPU clusters proposed in this invention is as follows: Figure 4 As shown. First, for neural network applications, each type of GPU is used to perform layer-by-layer characterization, establishing a cumulative distribution function model. Then, based on the physical network topology of the GPU and the topology of the neural network, a partitioning and task placement algorithm is designed, aiming to minimize the maximum computation and communication time of each stage after partitioning. Finally, based on the above partitioning and placement algorithm, pipelined distributed training is performed after task placement.

[0032] In the model characterization stage, this invention characterizes various data metrics of different GPUs under corresponding model and batch parameters in a heterogeneous GPU environment, providing them for subsequent modules. This stage constructs two modules, `torch_summary` and `torch_profile`, to characterize the model, describing the number of parameters, intermediate results, and computation time of each layer of a given model. `torch_summary` is responsible for characterizing the number of parameters and intermediate results. The number of parameters in a neural network is fixed, while the intermediate result values ​​are only related to the batch size. Therefore, given a batch size, it is only necessary to analyze the input-output tensors and parameter shapes of each layer of the neural network within this module to obtain the parameter values ​​and intermediate results of each layer. `torch_profile` is responsible for characterizing the computation time of each layer of the neural network. This requires training the neural network in a real environment and collecting relevant data. This invention modifies and recompiles the PyTorch source code to calculate the backpropagation time of each layer. Through the model characterization module, the number of parameters, intermediate results, and computation time of each layer of the model can be obtained, and these results are then used as the parameter input for the next layer of the model.

[0033] Before proceeding with the next step of partitioning the model within a node, this invention first models the pipeline partitioning optimization objective.

[0034] 1) Physical resource modeling in GPU clusters:

[0035]

[0036] Table 1 Results of Physical Resource Modeling

[0037] 2) Modeling the computational requirements of neural network models:

[0038]

[0039] Table 2. Results of Neural Network Model Computational Requirements Modeling

[0040] 3) Modeling in the model stage:

[0041]

[0042] Table 3. Modeling results at the model stage

[0043] 1) Physical Resource Modeling: In a GPU cluster, GPUs within a node are typically homogeneous, while GPUs across different nodes may be heterogeneous. Furthermore, the bandwidth within a node is homogeneous, but the internal bandwidth of different nodes may differ, and the bandwidth between different nodes may also differ. Therefore, GPUs are partitioned... The hierarchical structure of layers. Depend on The components are divided into parts, and these parts are used as the dividing units when dividing into stages. We use GPUs as the unit of division. We define components with the same computing power and bandwidth as the same type.

[0044] 2) Neural network model modeling: For model training in heterogeneous GPU clusters, the parameters and intermediate results of the neural network model are fixed after the hyperparameters and model are given. However, due to the differences in computing power and computation time caused by the differences in computing power of different types of GPUs, the neural network model needs to be characterized separately for each type of GPU.

[0045] 3) Model stage partitioning: The final result of model partitioning is to divide a complete model into several stages and decide whether to use data parallelism within these stages, while pipeline parallelism is executed between stages, while ensuring load balancing between stages to speed up training.

[0046] In a hybrid parallel pipeline mode, stages The training time for each round consists of two parts: the time for parallel data computation within each phase. Inter-stage data transmission communication time The data for parallel computation within a stage can be obtained from the computation time and image processing time of each layer of the neural network model, and can be represented as follows:

[0047]

[0048] Inter-stage data transmission communication time The communication operation consists of two parts: the first part is the time to receive intermediate results from the previous stage during forward propagation, and the second part is the time to receive gradients from the next stage during back propagation. In pipelined parallel mode, these two communication operations can be executed in parallel, thus reducing the communication time. You can take its maximum value:

[0049]

[0050] in, For the stage -1 to stage The size of intermediate results transmitted between nodes.

[0051] Furthermore, because computational and communication operations can also be executed in parallel in pipelined parallel mode, for each stage... Its training time This refers to the time required for parallel computation of data within a given stage. Inter-stage data transmission communication time The maximum value determines this, that is:

[0052]

[0053] This invention transforms the optimization objective, namely minimizing the overall model training time, into minimizing the training time of each round. Furthermore, since the overall training time is determined by the training time of the slowest stage in a pipelined parallel mode, the objective can be further transformed into minimizing the training time of the slowest stage, i.e.:

[0054]

[0055] After determining the optimization objective, this invention, adhering to the principle of using only homogeneous devices for data parallelism, and based on the hierarchical structure of real-world heterogeneous GPU clusters, first performs intra-node homogeneous GPU partitioning, then determines the partitioning points and selects devices for heterogeneous GPUs between nodes, and finally performs backtracking to obtain the required model partitioning and task placement scheme. The proposed pipelined hybrid parallel method for accelerating distributed neural network training in heterogeneous GPU clusters is described in the following pseudocode:

[0056]

[0057] The above algorithm is a model partitioning and task placement algorithm in a heterogeneous GPU cluster environment with a two-layer hierarchical structure. Main function The input is a GPU set The neural network model calculates the graph through the network model. , Bandwidth set within each node and bandwidth between nodes The output is the model partitioning and task placement structure. (First function) Used to solve The problem involves partitioning the model into layers, and the optimal partitioning point and the partitioning unit used will be updated during backtracking. The second function... Aimed at solving The optimal partitioning point, partitioning unit, and type are also updated during backtracking in the model partitioning problem. The third function... This is used to analyze the completed model partitioning results and execute a backtracking process to complete task placement. After completing model partitioning and task placement, the system will allocate model training tasks to the corresponding devices based on the scheme results and execute distributed pipelined parallel training.

[0058] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.

Claims

1. A pipelined parallel method for accelerating neural network training in heterogeneous GPU clusters, characterized in that, Includes the following steps: Step 1: Establish a hierarchical cumulative distribution function model for deep neural networks, analyze the corresponding input conditions required for the partitioning of deep learning application execution models and task placement schemes, and characterize the corresponding indicators of the neural network hierarchical structure based on the device topology by analyzing the model. Step 2: Based on the results of Step 1, for each type of GPU, a dynamic programming algorithm is used to divide the pipeline stages, and finally the optimal model partitioning and task placement scheme between any consecutive layers of the deep neural network trained by each type of GPU is obtained. Step 3: Based on the optimal model partitioning and task placement scheme obtained in Step 2, the results of Step 2 are used for partitioning on homogeneous GPUs within the node. A dynamic programming algorithm is then used to partition the pipeline stages between nodes using heterogeneous GPUs and heterogeneous network bandwidth, based on the partitioning within the node.

2. The pipelined parallel method for accelerating neural network training in a heterogeneous GPU cluster according to claim 1, characterized in that, In step 1, under pipelined mode, a cumulative distribution function model is established for the hierarchical structure of the neural network. Based on the cumulative distribution function model and the floating-point computing power of different types of GPUs, the metrics of each layer of the neural network during training are characterized when using each type of GPU.

3. The pipelined parallel method for accelerating neural network training in a heterogeneous GPU cluster according to claim 1, characterized in that, In step 2, the corresponding indicators obtained from step 1 are used as input, and a dynamic programming algorithm is used to solve for the training time when using this type of GPU to train any number of consecutive layers of a neural network.

4. The pipelined parallel method for accelerating neural network training in a heterogeneous GPU cluster according to claim 1, characterized in that, In step 3, based on steps 1 and 2, a dynamic programming algorithm is used to divide the model into multiple stages on different types of GPUs. The goal is to minimize the training time of the slowest stage to ensure load balance between stages.

5. The pipelined parallel method for accelerating neural network training in a heterogeneous GPU cluster according to claim 1 or 4, characterized in that, Includes the following sub-steps: Step 3-1: First, backtrack the results of Step 2 to obtain the optimal model partitioning and task placement scheme within the node for each type of GPU; Step 3-2: Next, based on the partitioning results within the node, a dynamic programming algorithm is used to classify and consider the GPU type used in this partitioning, compare the training time of the new partitioning stage, the bandwidth-based communication time of the partitioning stage, and the maximum value of the maximum time required by the remaining stages, so as to obtain the shortest training time when there are heterogeneous GPUs and heterogeneous bandwidths. Step 3-3: Finally, backtrack the dynamic programming results from Step 3-2 to obtain the optimal partitioning point, thereby obtaining the optimal model partitioning and task placement scheme.

6. The pipelined parallel method for accelerating neural network training in a heterogeneous GPU cluster according to claim 1 or 3, characterized in that, The dynamic programming algorithm process in step 2 is as follows: For the first layer topology The entire problem is broken down into several subproblems and the data transfer problem between adjacent subproblems. Only one subproblem is broken down at a time, which is the computation time of the data parallel phase. The recursive formula is expressed by the following formula: in Representing the The layer uses a type of The GPU performs parallel optimal time subproblems. Represents data communication time. The representative will The time required for data parallelism of a multilayer neural network using m GPUs of type t. The type used in this partitioning is The number of GPUs, The type used in this partitioning is The remaining number of GPUs; according to the above formula, traverse... Perform dynamic programming search to solve for the first-level topology. That is, the optimal solution for partitioning within a node.

7. The pipelined parallel method for accelerating neural network training in a heterogeneous GPU cluster according to claim 5, characterized in that, The dynamic programming algorithm process in step 3-2 is as follows: Heterogeneous GPUs are classified and discussed, and the recursive formula is as follows: in, for Layers, using The minimum training time per iteration after partitioning the GPU set into models is given. The right side of the equation represents various cases, each representing a data-parallel subproblem involving different types of GPUs. Indicate arrive Layers The types are The time required for data parallelism to be performed on the nodes, traversing You can get The optimal solution for partitioning. for The network bandwidth between nodes This represents the data communication time; after obtaining the optimal solution, it is necessary to backtrack the results to obtain the task placement scheme; the specific method is to first... The partitioning results are backtracked to obtain the optimal partition point and the GPU for the partition, i.e., the value in the formula. and Continue backtracking on the subproblems until no further partitioning is needed, and obtain the result. Task placement plan; Next, according to The task placement scheme will continue to be optimized. By backtracking, the final task placement result can be obtained.

8. The pipelined parallel method for accelerating neural network training in a heterogeneous GPU cluster according to claim 5, characterized in that, It also includes the following steps: Based on the model partitioning results, tasks are assigned to the corresponding GPUs, and the model code required for each stage of execution is generated for pipelined parallel computing.

Images