Large model automatic multi-dimensional parallel training system based on heterogeneous GPU

Abstract

The invention belongs to the technical field of deep learning model training, and particularly relates to a large model automatic multi-dimensional parallelization training system based on a heterogeneous GPU. The system comprises: (1) a 3D parallel composer which automatically generates a combination of data parallelism, tensor parallelism and pipeline parallelism to realize load balancing and efficient utilization of resources; (2) a parallel executor, in which a manager is placed, and communication overhead is reduced by optimizing physical mapping of a GPU (Graphics Processing Unit); the model divider is used for optimizing division of a model layer according to the performance of the GPUs and realizing load balance among different GPUs; the communication scheduler is used for realizing gradient synchronization among the asymmetric data parallel groups; and (3) a dynamic adapter which adopts a parameter-level check point recovery mechanism to quickly adjust and continue training when GPU resources are dynamically changed, so as to ensure the stability and continuity of the training process. Experimental results show that the method can effectively improve the training efficiency of the large model in the heterogeneous GPU environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning model training, and particularly relates to a large model automatic multi-dimensional parallelization training system based on heterogeneous GPUs. Background Technique

[0002] Deep Learning, as a subfield of machine learning, has made remarkable progress in recent years in multiple fields, driving the rapid development of artificial intelligence. Deep learning technology has demonstrated superior performance compared to traditional algorithms in application scenarios such as computer vision, speech recognition, natural language processing, and gaming. The Transformer model proposed by Vaswani et al. in 2017 abandoned the traditional recurrent neural network (RNN) structure and adopted the self-attention mechanism, greatly improving the efficiency of the model in processing long-sequence tasks. Models based on Transformer, such as BERT, GPT, etc., have achieved unprecedented success in tasks such as machine translation and text generation. As the scale of deep neural network models continues to expand, the computational resources and time required to train these models have increased exponentially. To address this issue, researchers have proposed various parallelization training methods, including data parallelism, pipeline parallelism, tensor parallelism, and hybrid parallelism. These methods improve training efficiency by distributing computational tasks to multiple GPUs for execution.

[0003] However, existing large model training frameworks usually assume that GPU resources are homogeneous, that is, all GPUs have the same computing power, video memory capacity, and communication bandwidth. Due to the rapid pace of GPU upgrades, there are significant differences in computing performance and video memory capacity between new and old models of GPUs. For example, NVIDIA's GPU models range from P100, V100, A100 to H100, with continuous improvements in computing performance and video memory capacity. At the same time, due to the high cost of GPU hardware, old models of GPUs are still in long-term use in clusters, resulting in an actual GPU cluster environment that is usually heterogeneous.

[0004] In addition, the development of cloud computing enables users to obtain GPU resources through cloud service providers. The preemptible instances provided by cloud service providers make the availability of GPU resources dynamic, and the number and type of GPUs may change over time. These factors pose the following challenges for training large-scale deep learning models in a heterogeneous GPU environment:

[0005] 1. Insufficient utilization of computing resources: Due to the different computing capabilities and video memory capacities of GPUs, how to effectively utilize the resources of heterogeneous GPUs has become a major problem. If the parallel strategy of homogeneous GPUs is adopted, it may lead to insufficient utilization of the computing resources of high-performance GPUs, while low-performance GPUs become the bottleneck of the system.

[0006] 2. Low communication efficiency: The communication bandwidth between heterogeneous GPUs may vary, and the physical location of GPUs (such as whether they are on the same node) will also affect communication efficiency. Existing parallel training frameworks usually do not optimize communication for heterogeneous environments, which may result in communication becoming the bottleneck of training.

[0007] 3. Inability to adapt to dynamic resource changes: In a cloud environment, GPU resources may change dynamically due to preemption or other reasons. Existing training frameworks usually do not support adjusting the parallel strategy during training, and are unable to adapt to the dynamic changes of GPU resources, resulting in training interruption or failure. Summary of the Invention

[0008] Aiming at the problems of insufficient utilization of computing resources, load imbalance, low communication efficiency, and inability to adapt to dynamic resource changes in the training of large-scale deep learning models in a heterogeneous GPU environment, the present invention proposes an automatic multi-dimensional parallel training system HetDawn for large models based on heterogeneous GPUs to achieve efficient training of large-scale deep learning models in a heterogeneous GPU environment.

[0009] The automatic multi-dimensional parallel training system HetDawn for large models based on heterogeneous GPUs provided by the present invention includes a 3D parallel orchestrator, a parallel executor, and a dynamic adapter group; among them:

[0010] The 3D parallel orchestrator automatically generates a combined strategy of data parallelism, tensor parallelism, and pipeline parallelism according to the model structure and the resource status of the heterogeneous GPU cluster, makes full use of the computing power and storage capacity of each GPU, and realizes load balancing and efficient resource utilization.

[0011] The parallel executor contains 3 sub-modules. Sub-module 1 is the placement manager, which, by optimizing the physical mapping of GPUs, preferentially assigns parallel operations with frequent communication between in-machine GPUs with high bandwidth to reduce communication overhead. Sub-module 2 is the model splitter, which optimizes the division of model layers according to the performance of GPUs to achieve load balancing between different GPUs. Sub-module 3 is the communication scheduler, which realizes gradient synchronization between asymmetric data parallel groups.

[0012] When the GPU resources change dynamically (for example, a preemptive instance is reclaimed or new resources are added), the dynamic adapter adopts a checkpoint recovery mechanism at the parameter level, quickly adjusts and continues training to ensure the stability and continuity of the training process.

[0013] Furthermore:

[0014] (1) The 3D parallel orchestrator defines which parallel group each GPU should be assigned to, and solves the load balancing of GPU grouping. According to the given model structure and heterogeneous GPU resources, an optimization model is established with the goal of maximizing the training efficiency and achieving load balancing. The objective function and constraints of the optimization model are as follows:

[0015] Objective function: Maximize the sum of the effective computing powers of all data parallel groups

[0016]

[0017] In the formula, x i,j represents whether the i-th GPU belongs to the j-th DP group, and y j is a binary value representing whether the j-th data parallel group is a valid data parallel group, that is, whether there is at least one GPU in the group. z represents the minimum value of the effective computing power in all data parallel groups. N represents the total number of data parallel groups.

[0018] The effective computing power value is defined as the actual computing power value presented by the data parallel group under multi-dimensional parallelism, and its calculation formula is as follows:

[0019]

[0020] In the formula, represents the computing power value of the i-th GPU, and ρ j represents the bubble rate in the j-th data parallel group.

[0021] The bubble rate is the idle ratio of computing devices brought about in the pipeline parallel mode, and its calculation formula is as follows:

[0022]

[0023] In the formula, PP j represents the number of pipeline stages in the j-th data parallel group, and k represents the number of micro-batch samples.

[0024] This optimization model satisfies the following constraints:

[0025] Constraint 1:

[0026] G j ·y j +L·(1 - y j ) ≥ z, (4)

[0027] In the formula, L represents a large number, and this constraint ensures that z is the minimum effective computing power value among all valid data parallel groups.

[0028] Constraint 2:

[0029]

[0030] In the formula, m i represents the video memory of the i-th GPU device, and Min mem represents the minimum video memory capacity required for running model training. This constraint ensures that the video memory capacity of all valid data parallel groups is greater than the minimum video memory capacity required for model operation.

[0031] Constraint 3:

[0032]

[0033] This constraint ensures that when there is at least one GPU in the j-th data parallel group, the value of y j is 1, otherwise it is 0.

[0034] (2) The parallel executor described above consists of the following three sub-modules: (1) The placement manager solves the challenge of GPU mapping by optimizing communication between parallelisms; (2) The model splitter proposes an optimized model to solve the challenge of model splitting and maximize balanced computing power; (3) The communication scheduler performs communication scheduling to solve the communication problem between asymmetric parallel structures.

[0035] The first sub-module is the placement manager. After given heterogeneous GPUs and the GPU grouping results, the placement manager decides: (1) Map GPUs to actual physical nodes; (2) Allocate pipeline stages for each GPU. The placement manager adopts a greedy strategy to solve the above challenges. The design concept of the algorithm follows the following two principles: (1) Allocate high-bandwidth (NVLink within the node) and low-bandwidth (RDMA between nodes) in the order of tensor parallelism, data parallelism, and pipeline parallelism. (2) Try to place GPUs with lower computing power in the earlier stages of pipeline parallelism. The input of the algorithm is the number of each type of card in each data parallel group, as well as the actual card type and quantity on each node. The output of the algorithm is the node arrangement and pipeline stage arrangement of each card in each pipeline parallel group.

[0036] Sub-module 2 is a model splitter, which solves the problem of load balancing of model layers caused by the heterogeneity of GPU computing power and video memory. The model splitter proposes an optimized model to effectively implement model segmentation. The design concept of the optimization algorithm is to allocate the number of model layers according to the computing power of each stage in pipeline parallelism during training, and the video memory capacity of each pipeline stage should not be exceeded, so as to make the iteration time of each stage as close as possible and reduce the waiting time between stages. The input of the algorithm is the node arrangement and pipeline stage arrangement of each card in each data parallel group. The output of the algorithm is the number of model layers that should be allocated to each type of card within each data parallel group.

[0037] The objective function and constraints of the optimized model are as follows:

[0038] Objective function:

[0039]

[0040] In the formula, P represents the number of stages in the data parallel group, and g i represents the computing power value of the GPU in the i-th stage of the data parallel group, and l i represents the number of model layers allocated to the i-th stage of the data parallel group. The meaning of this objective function is to minimize the maximum ratio of GPU computing power and allocated layers in the data parallel group.

[0041] Constraint 1:

[0042]

[0043] In the formula, L represents the total number of model layers. This constraint ensures that in this data parallel group, the sum of the number of model layers responsible for each pipeline stage is the total number of model layers.

[0044] Constraint 2:

[0045]

[0046] In the formula, F(·) represents the fixed video memory overhead when calculating the model operation, V(·) represents the dynamic video memory overhead when calculating the model operation, and m i represents the video memory size of the i-th GPU. This constraint ensures that the workload allocated to the i-th GPU does not exceed the video memory capacity of this GPU.

[0047] Sub-module 3 is a communication scheduler. In order to adapt to the communication in the asymmetric parallel structure, it is necessary to modify the standard All-reduce primitive. This system designs a communication scheduler that bypasses the traditional "stage" concept between data parallel groups, introduces a finer-grained hierarchical communication framework, and abstracts virtual data parallel groups to implement the asymmetric All-reduce primitive.

[0048] The virtual data parallel group is defined as a collection of GPUs from different data parallel groups that jointly manage the same consecutive model layers. The GPUs generate corresponding model layers based on the allocated number of layers and their positions in the pipeline stage. For example, if GPU-0 is allocated 4 layers and occupies the first pipeline stage, it will be responsible for layers 0-3. The All-reduce primitive can effectively synchronize gradients when executed within each virtual data parallel group because the GPUs within the virtual communication group share the gradients of the same model layer.

[0049] The communication scheduler initializes multiple virtual communication groups. In each virtual communication group, at least one model layer gradient is required for All-reduce communication. Since each GPU stores the gradients of all parameters in the form of a one-dimensional vector in layer order, the communication scheduler extracts gradient slices for All-reduce communication based on the specified consecutive layer indices in the current virtual communication group.

[0050] (3) The dynamic adapter adopts a parameter-level checkpoint method and introduces a checkpoint acceleration mechanism. The dynamic adapter first saves model parameters, optimizer states, etc. as a single.pth file at the granularity of model layers. If the dimension of tensor parallelism remains the same in two consecutive parallelization methods, each GPU in the new parallelization method can directly load the checkpoint according to the model layers allocated to it. If the dimension of tensor parallelism changes between the two consecutive parallelization methods, then during checkpoint reading, each parameter is concatenated or disassembled to adapt to the current tensor parallelism dimension.

[0051] To relieve the pressure on cloud bandwidth, the system adopts a layer-bit map method to match the parameters required in the new changed parallelization method with the checkpoint owned by the current node. If available locally, it is directly read from the local; otherwise, the unowned parameters are pulled from the cloud.

[0052] The present invention mainly has the following advantages:

[0053] (1) Automatically generate the optimal parallelization: Through the 3D parallel orchestrator, an optimization model is established based on the computing power, video memory capacity, and physical location of the GPUs to automatically generate the optimal parallelization strategy, including a combination of data parallelism, tensor parallelism, and pipeline parallelism. Ensure that in a heterogeneous GPU environment, the computing tasks of each GPU match its capabilities and achieve load balancing.

[0054] (2) Efficiently execute parallel strategies: Through a parallel executor, perform the segmentation of the model and the mapping to GPUs, and schedule the communication between GPUs. The mapping manager, based on the communication priority, preferentially arranges high-frequency and large-volume communications on the GPUs connected by high-speed in-machine bandwidth, optimizing the communication efficiency. The model splitter, based on the optimized model, splits each layer of the model onto different GPUs to achieve load balancing. The communication scheduler introduces a more fine-grained hierarchical communication framework for the asymmetric parallel structure, solving the communication problem under heterogeneous GPUs.

[0055] (3) Support elastic training for dynamic resource changes: Through a dynamic adapter, adopt a checkpoint method based on parameter levels to achieve training recovery under dynamic changes in GPU resources. Ensure that when the parallelism and GPU resources change, the training state can be correctly restored, reducing the overhead caused by training interruptions and improving the continuity and reliability of training.

[0056] (4) Improve training efficiency and reduce training costs: The present invention can make full use of the computing and storage resources of heterogeneous GPUs, improve the training efficiency of large models, shorten the training time, reduce the training costs, and is applicable to large-scale deep learning model training scenarios that require high-performance computing resources. Description of the Drawings

[0057] Figure 1 It is an architecture diagram of an automatic parallelization system for heterogeneous training.

[0058] Figure 2 It is a throughput test diagram under a consistent GPU distribution.

[0059] Figure 3 It is a throughput test diagram under a non-consistent GPU distribution. Detailed Implementation Manner

[0060] The design of an automatic multi-dimensional parallelization training system for large models based on heterogeneous GPUs specifically includes the following steps:

[0061] Step 1: The heterogeneous GPU multi-dimensional parallel training system of the present invention calls a 3D parallel orchestrator to start constructing a combination of parallel strategies in a heterogeneous GPU environment. This includes automatically generating a combined strategy of data parallelism, tensor parallelism, and pipeline parallelism according to the model structure and the resource status of the GPU cluster; making full use of the computing power and storage capacity of each GPU to achieve load balancing and efficient resource utilization. This step relies on the 3D parallel orchestrator to solve the problems of GPU grouping and load balancing.

[0062] Step 2: After the parallel strategy is generated, the parallel executor starts to perform model segmentation, GPU mapping, and communication scheduling. When communication and synchronous gradient data updates need to be optimized, three sub-modules of the parallel executor are called: The placement manager optimizes the physical mapping of GPUs, preferentially allocating parallel operations with frequent communication among on-board GPUs with high bandwidth to reduce communication overhead; the model splitter optimizes the division of model layers according to the performance of GPUs to achieve load balancing among different GPUs; the communication scheduler realizes gradient synchronization between asymmetric data parallel groups to ensure the coordinated operation of each GPU.

[0063] Step 3: A dynamic adapter is designed for the dynamic changes of GPU resources. When GPU resources change (e.g., preemptive instances are reclaimed or new resources are added), the dynamic adapter adopts a checkpoint recovery mechanism at the parameter level to quickly adjust and continue training, ensuring the stability and continuity of the training process. It first saves the model parameters and optimizer states, and then reloads and allocates these parameters according to the new GPU resource situation to ensure that the training does not interrupt.

[0064] Step 4: During the training process, the target GPU node continuously listens for and executes training tasks, and monitors the dynamic changes of GPU resources in real time. When resource changes are detected, the dynamic adapter is immediately started to adjust the parallel strategy, and the layer-bitmap method is used to match the newly required parameters with the checkpoints owned by the current node, reducing the overhead of pulling data from the cloud and ensuring the continuity and efficiency of the training process.

[0065] Step 5: If there is an increase or decrease in GPU resources during the training process, the dynamic adapter adjusts the division of model layers and the mapping of GPUs by reconfiguring the parallel strategy to ensure load balancing among GPUs. The communication scheduler simultaneously adjusts the communication plan to adapt to the new parallel structure, avoiding communication bottlenecks and improving training efficiency.

[0066] Step 6: After the training task is completed, the system is properly shut down to ensure the stability of the system.

[0067] Experimental example:

[0068] The experimental environment consists of four GPU server nodes, equipped with a total of 32 GPUs. The specific cluster configuration is as follows: Nodes 1 and 2 are each equipped with 8 NVIDIA A100 GPUs, each with 80GB of video memory; Node 3 is equipped with 8 NVIDIA H800 GPUs, also with 80GB of video memory; Node 4 is equipped with 8 NVIDIA H20 GPUs, with a video memory capacity of 100GB. High-speed communication is carried out using NVLink inside the nodes, with a bandwidth range between 400 and 600GB / s; Communication between nodes is achieved through an RDMA network, with a bandwidth of 400Gb / s. Three model architectures, namely Bert-Large, GPT-3, and LLaMA, are selected.

[0069] Figure 2 Shown is the system performance under the consistent GPU distribution experiment settings, with two model combinations of A100+H800 and A100+H20 selected, and each node containing 2, 4, or 8 GPUs. In the experiment with the BERT-large 340M model architecture, HetDawn's average performance is 1.38 times better than Megatron-LM. Since the BERT-large model has a relatively small number of parameters, it means that all its storage requirements such as model parameters, optimizer states, forward activation values, etc. can be fully stored on a single GPU. Therefore, the optimal parallel strategy for Megatron-LM is to use full data parallelism. However, this parallel strategy results in a significant stragger phenomenon. In contrast, Whale adjusts the batch size according to the computing power of the GPUs, enabling it to achieve effective load balancing under this experimental setting. HetDawn, on the other hand, uses a combination of data parallelism and pipeline parallelism, and the automatically generated parallel strategy can achieve almost the same performance without affecting convergence.

[0070] In the experiments on the GPT-3 6.7B model structure, HetDawn's performance outperformed Megatron-LM and Whale, with an average improvement of 1.53 times and 1.27 times respectively. The performance improvement can be attributed to two main factors. First, different from the experiments on the above BERT-large model, the model storage requirements of the GPT-3 structure cannot be met by a single GPU, and a model parallelism strategy must be adopted. HetDawn's workload balancing method takes into account GPU heterogeneity and can reasonably allocate model layers according to the computing capabilities of different GPUs, thus making full use of the computing resources of different GPUs. In contrast, Megatron-LM assumes that GPUs are homogeneous and adopts a method of evenly dividing model layers, which fails to take into account the differences in GPU performance. Second, HetDawn's hybrid parallel communication optimization strategy schedules low-computing-power GPU models to the front pipeline stages, effectively balancing the trade-off between computing power and memory utilization. Megatron-LM and Whale, on the other hand, only allocate pipeline stages according to the order of GPU nodes, limiting their optimization ability of resource utilization.

[0071] Figure 3 The figure shows the system performance under non-uniform GPU distribution. The experiment selected the GPU model combination of A100 + H800 and the model structure of LLaMA 6.7B. In the experiment, the numbers of A100 and H800 were controlled to be 2 and 4, 4 and 2, as well as 3 and 5, 5 and 3, corresponding to the cases where the effective TP dimension in the setting was 2 or 1 respectively. HetDawn significantly outperformed Megatron-LM and Whale, with improvements of 1.79 times and 1.51 times respectively. One of the main reasons for HetDawn's performance improvement under non-uniform GPU distribution is that it supports an asymmetric parallel structure, while the baseline systems are limited by symmetric parallelism. Therefore, HetDawn can construct a more computing-power-balanced non-uniform GPU device group than Megatron-LM and Whale. As Figure 3 (a) shows, HetDawn can configure the TP degree to be 2, where the H800 models form a separate DP group, and the A100 models form another DP group with a two-stage pipeline. In contrast, Megatron-LM and Whale can only form a more costly long-pipeline parallel structure. Similarly, in Figure 3 (c) and Figure 3 (d), when the number of GPUs per node is odd and a TP group cannot be formed, HetDawn can still maintain load balance through a more flexible combination of pipeline and data parallelism. In contrast, Megatron-LM and Whale cannot achieve load balance of model layers between different DP groups because they do not support configurations with inconsistent numbers of layers within the same stage.

[0072] References:

[0073] [1]Deepak Narayanan,Aaron Harlap,Amar Phanishayee,Vivek Seshadri,Nikhil R Devanur,Gregory R Ganger,Phillip B Gibbons,andMateiZaharia.2019.PipeDream:

[0074] Generalized pipeline parallelism for DNN training.In Proceedings ofthe 27thACM symposium on operating sys-tems principles.1–15.

[0075] [2]Deepak Narayanan,Mohammad Shoeybi,Jared Casper,Patrick LeGres-ley,MostofaPatwary,Vijay Korthikanti,Dmitri Vainbrand,PrethviKashinkunti,JulieBernauer,Bryan Catanzaro,et al.2021.Efficient large-scale languagemodeltraining on gpu clusters using megatron-lm.In Proceedings of theInternationalConference for High Performance Computing,Networking,Storage andAnalysis.1–15。

Claims

1. An automatic multi-dimensional parallel training system HetDawn for large models based on heterogeneous GPUs, characterized in that, It includes a 3D parallel orchestrator, a parallel executor, and a dynamic adapter group; among which: The 3D parallel orchestrator automatically generates a combined strategy of data parallelism, tensor parallelism, and pipeline parallelism according to the model structure and the resource status of the heterogeneous GPU cluster, making full use of the computing power and storage capacity of each GPU to achieve load balancing and efficient resource utilization; The parallel executor contains 3 sub-modules: Sub-module 1 is the placement manager, which, by optimizing the physical mapping of GPUs, preferentially allocates parallel operations with frequent communication among the on-board GPUs with high bandwidth to reduce communication overhead; Sub-module 2 is the model splitter, which optimizes the division of model layers according to the performance of GPUs to achieve load balancing among different GPUs; Sub-module 3 is the communication scheduler, which realizes gradient synchronization between asymmetric data parallel groups; The dynamic adapter, when the GPU resources change dynamically, adopts a checkpoint recovery mechanism at the parameter level to quickly adjust and continue training to ensure the stability and continuity of the training process.

2. The large model automatic multi-dimensional parallel training system HetDawn based on heterogeneous GPUs according to claim 1, wherein The 3D parallel orchestrator defines which parallel group each GPU should be assigned to, so as to balance the GPU grouping load; according to the given model structure and heterogeneous GPU resources, an optimization model is established, and the goal is to maximize the training efficiency and achieve load balancing; the objective function and constraints of the optimization model are as follows: Objective function: Maximize the sum of the effective computing powers of all data parallel groups: where x i,j represents whether the i-th GPU belongs to the j-th DP group, and y j is a binary value representing whether the j-th data parallel group is a valid data parallel group, that is, whether there is at least one GPU in the group; z represents the minimum value of the effective computing power in all data parallel groups, and N represents the total number of data parallel groups; The effective computing power value is defined as the actual computing power value presented by the data parallel group under multi-dimensional parallelism, and its calculation formula is as follows: where, g i represents the computing power value of the i-th GPU, and ρ j represents the void fraction in the j-th data parallel group; The void fraction is the idle ratio of computing devices brought about in the pipeline parallel mode, and its calculation formula is as follows: wherein, PP j represents the number of pipeline stages in the j-th data parallel group, and t represents the number of micro-batch samples; This optimization model satisfies the following constraints: Constraint 1: G j ·y j +L·(1 - y j ) ≥ z, (4) In the formula, L represents a large number, and this constraint ensures that z is the smallest effective computing power value among all effective data parallel groups; Constraint 2: where m i represents the video memory of the i-th GPU device, and Min mem represents the minimum video memory capacity required for running model training. This constraint ensures that the video memory capacity of all valid data parallel groups is greater than the minimum video memory capacity required for model operation; Constraint 3: This constraint ensures that when there is at least one GPU in the j-th data parallel group, the value of y j is 1, otherwise it is 0.

3. The large model automatic multi-dimensional parallel training system HetDawn based on heterogeneous GPUs according to claim 2, wherein In the parallel executor: The placement manager, after given the heterogeneous GPUs and the GPU grouping results, needs to decide: (1) Map the GPUs to the actual physical nodes; (2) Allocate pipeline stages for each GPU; the placement manager uses a greedy strategy to handle it; Its algorithm follows the following two principles: (1) Allocate high bandwidth in turn according to the priorities of tensor parallelism, data parallelism, and pipeline parallelism; (2) Try to make the GPUs with lower computing power be in the earlier stages of pipeline parallelism; the input of the algorithm is the number of each type of card in each data parallel group and the actual card type and number on each node; the output of the algorithm is the node arrangement and pipeline stage arrangement of each card in each pipeline parallel group; The model splitter uses an optimization model to achieve effective segmentation of the model; it is required that in pipeline parallelism, the number of model layers is allocated according to the computing power of each stage, and the video memory capacity of each pipeline stage is not exceeded during training, so that the iteration time of each stage is as close as possible; the input of the algorithm is the node arrangement and pipeline stage arrangement of each card in each data parallel group; the output of the algorithm is the number of model layers that should be allocated to each type of card within each data parallel group; The objective function and constraints of the optimization model are as follows: Objective function: where P represents the number of stages in the data parallel group, and g i represents the computing power value of the GPU at the i-th stage in the data parallel group, and l i represents the number of model layers allocated to the i-th stage in the data parallel group; the meaning of this objective function is to minimize the maximum ratio of GPU computing power to the allocated number of layers in the data parallel group; Constraint 1: In the formula, L represents the total number of layers of the model; this constraint ensures that in this data parallel group, the sum of the number of model layers responsible for each pipeline stage is the total number of layers of the model; Constraint 2: Wherein, F(·) represents the fixed video memory overhead during the operation of the computing model, V(·) represents the dynamic video memory overhead during the operation of the computing model, and m i represents the video memory size of the i-th GPU; this constraint ensures that the workload allocated to the i-th GPU does not exceed the video memory capacity of this GPU; The communication scheduler modifies the standard All-reduce primitive to adapt to the communication in the asymmetric parallel structure; specifically, it bypasses the traditional "stage" concept between data parallel groups, introduces a finer-grained hierarchical communication framework, and abstracts virtual data parallel groups to implement the asymmetric All-reduce primitive; The virtual data parallel group is defined as a set of GPUs from different data parallel groups, and these GPUs jointly manage the same continuous model layers; the GPU generates corresponding model layers according to the allocated number of layers and its position in the pipeline stage; the All-reduce primitive effectively synchronizes gradients when executed within each virtual data parallel group; The communication scheduler initializes multiple virtual communication groups; in each virtual communication group, the All-reduce communication requires at least one model layer gradient; since each GPU stores the gradients of all parameters in the form of a one-dimensional vector in layer order, the communication scheduler extracts gradient slices according to the specified continuous layer index in the current virtual communication group for All-reduce communication.

4. The large model automatic multi-dimensional parallel training system HetDawn based on heterogeneous GPUs according to claim 3, characterized in that The dynamic adapter adopts a parameter-level checkpoint method and introduces a checkpoint acceleration mechanism; first, it saves the model parameters and optimizer state as a.pth file at the granularity of model layers; If the dimension of tensor parallelism remains the same in the two parallel modes before and after, each GPU in the new parallel mode directly loads the checkpoint according to the model layers allocated to it; if the dimension of tensor parallelism changes in the two parallel modes before and after, then when reading the checkpoint, each parameter is concatenated or disassembled to adapt it to the current tensor parallel dimension; To relieve the pressure on the cloud bandwidth, a layer-bit map method is adopted to match the parameters required in the new changed parallel mode with the checkpoint owned by the current node. If it is available locally, it is directly read from the local, otherwise the unowned parameters are pulled from the cloud.

Images