A large model hybrid parallel training method and system for heterogeneous environments

Abstract

The application provides a large model hybrid parallel training method and system for a heterogeneous environment, based on the computing power of different data centers in the heterogeneous environment and the network delay between the different data centers, taking the computing power balance of the data centers in each training stage of the to-be-trained model and the equality of the computing task transmission of the nodes contained in the data centers in each training stage as the target, using a genetic algorithm for iterative optimization to obtain the data centers corresponding to each training stage and the computing tasks of different nodes in each training stage, training the to-be-trained model, thereby reducing the computing cost caused by resource balance and cross-domain, and improving the large model training efficiency.

Description

Technical Field

[0001] This invention belongs to the field of model parallel training technology, and in particular relates to a method and system for hybrid parallel training of large models in heterogeneous environments. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, large-scale pre-trained models have made rapid progress, becoming a hot topic in international cutting-edge artificial intelligence research and applications. In 2021, Stanford University established the Center for Foundation Models (CRFM), collectively referring to models such as BERT, GPT-3, and CLIP, which are trained on large-scale data and applicable to multiple downstream tasks, as Foundation Models. OpenAI released the large-scale natural language pre-trained model GPT-3, as well as the multimodal pre-trained models DALL-E and CLIP, while Google released the Switch Transformer, a pre-trained language model with 1.6 trillion parameters. With the increasing scale of pre-trained models, the cost and time of training have increased significantly. However, due to limited resources and numerous demands on various supercomputing platforms, a single data center cannot support the training of large-scale pre-trained models. Therefore, combining computing power from multiple locations and collaboratively completing the training of large-scale pre-trained models based on a supercomputing internet has become a key technology that needs to be addressed.

[0004] Traditionally, training large models in homogeneous environments involves pipelined parallelism to partition the model and scaling up through data parallelism. While pipelined parallelism is effective for communication in homogeneous environments, it's insufficient on its own. In heterogeneous environments, because each participant in a group has different computational and networking capabilities, the pipeline formed by these devices will be dominated by a single weakest link—the participant with the lowest training throughput. As a result, more powerful nodes along the pipeline will be underutilized due to lack of input or slow subsequent stages. Furthermore, in heterogeneous environments, data parallelism requires cross-domain communication for gradient synchronization, which also increases the time cost of training the model.

[0005] In summary, existing technologies suffer from the following problems: In a dynamic heterogeneous environment, pipelined parallelism and data parallelism training exhibit very poor tolerance for heterogeneous networks. Furthermore, heterogeneous environments contain different hardware devices with varying computing capabilities; assigning tasks according to a fixed schedule can lead to issues such as high-performance computing nodes being idle or low-performance computing nodes experiencing memory overflow. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this invention provides a hybrid parallel training method and system for large models in heterogeneous environments. By considering the computing power of different data centers in heterogeneous environments, the network latency between different data centers, and the equivalence of computing tasks of different nodes in the same training phase, the invention optimizes the training of the model to be trained by using a genetic algorithm.

[0007] To achieve the above objectives, a first aspect of the present invention provides a hybrid parallel training method for large models in heterogeneous environments, comprising:

[0008] Obtain the computing power of different data centers in a heterogeneous environment and sort the network latency between different data centers;

[0009] Based on the ranking of network latency among different data centers, the two data centers with the highest network latency are assigned to the first and last training stages of the pipeline parallel training of the model to be trained.

[0010] Based on the pipeline parallel scale of the model to be trained, with the goals of balanced computing power of the data center between each training stage and equivalence of computing task transmission among the nodes contained in the data center in each training stage, the genetic algorithm is used for iterative optimization to obtain the data center corresponding to each training stage and the computing tasks of different nodes in each training stage.

[0011] The model to be trained is trained based on the data center corresponding to each training stage and the computational tasks of different nodes within each training stage.

[0012] A second aspect of the present invention provides a large-scale model hybrid parallel training system for heterogeneous environments, comprising:

[0013] The acquisition module is used to acquire the computing power of different data centers in a heterogeneous environment and sort the network latency between different data centers.

[0014] The allocation module is used to assign the two data centers with the highest network latency to the first and last training stages of the pipeline parallel training of the model to be trained, based on the ranking results of network latency between different data centers.

[0015] Optimization module: Based on the pipeline parallel scale of the model to be trained, with the goals of balanced computing power of data centers between each training stage and equivalence of computing task transmission among nodes in each training stage, iterative optimization is performed using a genetic algorithm to obtain the data center corresponding to each training stage and the computing tasks of different nodes within each training stage.

[0016] Training module: Based on the data center corresponding to each training stage and the computing tasks of different nodes within each training stage, the model to be trained is trained.

[0017] A third aspect of the present invention provides a computer device comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and the machine-readable instructions, when executed by the processor, perform a large-model hybrid parallel training method for heterogeneous environments.

[0018] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs a hybrid parallel training method for large models in heterogeneous environments.

[0019] The above one or more technical solutions have the following beneficial effects:

[0020] In this invention, based on the computing power of different data centers in a heterogeneous environment and the network latency between them, the goal is to achieve computing power balance among data centers in each training stage of the model to be trained and equivalence of computational task transmission among nodes in each training stage. A genetic algorithm is used for iterative optimization to obtain the data centers corresponding to each training stage and the computational tasks of different nodes within each training stage. The model to be trained is then trained, thereby reducing the computational costs associated with resource balancing and cross-domain operations, and improving the training efficiency of large models. Furthermore, this method allows for flexible use of devices in heterogeneous environments, enabling collaborative training of a large model based on wide-area computing power. Moreover, since the first and last stages of the pipeline do not communicate, using the two data centers with the highest network latency as the first and last stages of the parallel pipeline can reduce the overall cost of model training.

[0021] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 This is a flowchart of the large-scale hybrid parallel training method for heterogeneous environments in Embodiment 1 of the present invention.

[0024] Figure 2 This is a schematic diagram of the distributed training architecture in a heterogeneous environment constructed in Embodiment 1 of the present invention. Detailed Implementation

[0025] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0027] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0028] Example 1

[0029] like Figures 1-2 As shown, this embodiment discloses a hybrid parallel training method for large models in heterogeneous environments, including:

[0030] Obtain the computing power of different data centers in a heterogeneous environment and sort the network latency between different data centers;

[0031] Based on the ranking of network latency among different data centers, the two data centers with the highest network latency are assigned to the first and last training stages of the pipeline parallel training of the model to be trained.

[0032] Based on the pipeline parallel scale of the model to be trained, with the goals of balanced computing power of the data center between each training stage and equivalence of computing task transmission among the nodes contained in the data center in each training stage, the genetic algorithm is used for iterative optimization to obtain the data center corresponding to each training stage and the computing tasks of different nodes in each training stage.

[0033] The model to be trained is trained based on the data center corresponding to each training stage and the computational tasks of different nodes within each training stage.

[0034] This embodiment replaces the fixed pipeline structure with a flexible pipeline structure based on different computing power scales. Each participant can send its output to any peer serving the next pipeline stage. Therefore, if a peer has significantly more computation than others, it can process inputs from multiple predecessors and distribute its output across several weaker peers to maximize utilization. Furthermore, this approach maximizes data parallelism within the same data center, synchronously optimizing gradients in heterogeneous environments. The hybrid parallel training method for large models in heterogeneous environments proposed in this embodiment can effectively utilize computing power from different locations to collaboratively train large models while reducing training costs.

[0035] The hybrid parallel training method for large models in heterogeneous environments proposed in this embodiment specifically includes:

[0036] Step 1: In a heterogeneous environment, determine the existing computing power of each local data center and the latency and bandwidth between them;

[0037] Specifically, different data centers are treated as integrated computing units. These data centers may be distributed in different geographical locations, so actual end-to-end communication will suffer from latency issues due to bandwidth and distance. The network latency between different data centers is ranked to determine the interaction latency between each data center and other data centers; the two data centers with the highest latency are selected and initialized as the beginning and end of a pipeline in parallel.

[0038] Specifically, different data centers are treated as a whole computing power unit. A node is selected from each data center as the representative of the network latency and bandwidth of that center. The IP of each representative node and the latency and bandwidth information between it and other nodes are written into a configuration file, and the topology is composed of all the representative nodes.

[0039] The network latency between different data centers is sorted. Specifically, this involves finding the lowest latency by distance among the various data centers, and the Floyd-Warshall algorithm is used to sort the data centers. The Floyd-Warshall formula is shown below:

[0040] D i,j,k =min(D i,j,k-1 D i,k,k-1 +D k,j,k-1 (1)

[0041] D i,j,k =D i,k,k-1 +D k,j,k-1 (2)

[0042] D i,j,k =D i,k,k-1 (3)

[0043] Where i, j, and k represent the lengths of the shortest path between node i and node j in k iterations. The Floyd-Warshall algorithm is applicable to the shortest path problem between all nodes and can find the latency between any two data centers.

[0044] Step 2: Determine the computing tasks to be undertaken by each data center and each specific node based on the computing power scale of different data centers and the computing capabilities of different devices within the data centers. Data centers with larger computing power can undertake more computing tasks, i.e., be allocated into multiple Stages; while data centers with smaller computing power can be aggregated into a stage of a pipeline, i.e., combined into a Stage.

[0045] Employing the 1F1B algorithm as the pipeline parallelism strategy effectively reduces pipeline idle waiting time. Based on the user-input pipeline size parameters, a global pipeline queue is created, dividing the entire model into several stages. A queue is created for each stage to store nodes performing data parallelism. The first and last nodes identified by Floyd-Warshall represent the nodes with the highest communication costs; direct communication could lead to latency. Since the first and last stages of the pipeline do not communicate, the first and last stages include the two data centers with the highest latency, thus avoiding communication and reducing the overall model training time and cost.

[0046] In this embodiment, grouping based on latency and bandwidth is actually accomplished using a genetic algorithm, with latency, bandwidth, and computing power as input parameters. After grouping, multiple Stage queues are output, each containing a different node ID; for example, Stage1: [0,1,2,6,7]. Here, nodes 0, 1, and 2 might be three nodes from supercomputing center A, and nodes 6 and 7 might be nodes from location B. In this way, the task of each node is determined, i.e., which nodes handle which stage of pipeline parallelism, and which nodes handle which data parallel task.

[0047] The grouping based on delay bandwidth is actually accomplished using a genetic algorithm, specifically:

[0048] Initialize 100 sequences based on all nodes, and perform 4900 optimizations. The number of initializations and optimizations can be customized.

[0049] Each sequence can be replaced with a solution, which is the size of the data parallel groups and pipeline parallel groups that need to be divided. For example, 10*5 means 10 stages and 5 data parallel groups in each stage.

[0050] After dividing into solutions, the optimal combination is determined by using the formula with delay bandwidth as the input parameter.

[0051] Data parallelism: Each device needs to send each of its local fragments to the corresponding device; next, each device can aggregate the gradients it receives from all other devices; finally, each device sends the aggregated gradient fragments to all other devices. Therefore, the cost of data parallelism is actually the latency between the two locations + the gradients that need to be exchanged between individual devices (total exchange volume / total number of devices) / bandwidth between the two locations. Thus, latency and bandwidth are incorporated into the optimization strategy.

[0052] Pipeline parallelism is similar, except that it transmits forward propagation activation values ​​and backward propagation gradient values.

[0053] Then, the 4900 optimization from the genetic algorithm is used to find the optimal cost:

[0054] 1) The input is a set of candidate equilibrium graph partitions used as the initial population. The algorithm generates the next generation as follows;

[0055] 2) It first generates a new "descendant" o, given two randomly selected "parents" p1 and p2;

[0056] 3) A popular approach is to randomly swap some nodes between the two parent nodes. Given this descendant o, a local search is then performed starting from o to find a new balanced partitioning strategy o*, which yields better cost.

[0057] 4) Then, add o* to the population, and remove the worst partition candidate from the population if o* has a better cost.

[0058] Then, the process is repeated 4900 times to find the best combination and add it to the optimal situation of this sequence.

[0059] It's important to understand that although the data center is treated as a whole, the genetic algorithm is actually executed node by node. Therefore, the final output is also nodes, and the allocation is done between nodes. There's no need to consider the data center issue.

[0060] Specifically, based on the computing power of different data centers, the amount of computing tasks each data center undertakes in pipeline parallelism is determined.

[0061]

[0062]

[0063] Wherein, Node_Power represents the computing power of each location as a data center, and the total computing power scale all_computility in a wide-area heterogeneous environment is calculated by formula (4); pipeline_parallel_size represents the scale of pipeline parallelism, and the computing power resources stage_task allocated to each Stage task are calculated by formula (5) based on the scale of pipeline parallelism.

[0064] Understandably, the parallelism scale of a pipeline is the number of stages. The number of stages is generally determined by hyperparameters input by the user, and the total computing power needs to be considered. If it is too small, memory overflow issues will occur.

[0065] Meanwhile, in a heterogeneous environment, each data center includes different resources and equipment. Flexible pipeline structures are built based on these different devices, allowing each node to send its output to any peer serving the next stage of the pipeline. Therefore, if a peer has significantly more computation than others, it can handle input from multiple previous generations and distribute its output across several weaker peers to maximize utilization. For example, an A100 graphics card, with more video memory and stronger computing power than a T4 graphics card, will handle more computational tasks.

[0066] By adopting data parallelism to expand the scale of distributed training of models, the synchronization in data parallelism in heterogeneous environments is transformed as much as possible into homogeneous environments. Nodes within each data center can perform gradient synchronization faster, reducing the additional overhead caused by latency and bandwidth.

[0067] Data parallelism is a common strategy for scaling up model training. However, in heterogeneous environments, model synchronization between data parallel nodes is highly sensitive to the slowest worker. A delay in one node impacts both data parallelism efficiency and the idle waiting overhead of all other stages and pipelines, leading to low overall training efficiency. To address this issue, each data center is treated as a single unit, reducing the need for data center splitting. Different data centers are combined and aggregated, employing a flexible approach to tolerate heterogeneity. An all-reduce synchronization strategy is used, with each node communicating with each other in a point-to-point manner. This results in more even communication distribution and ensures convergence of the data parallel model.

[0068] Step 3: Using the planned allocated computing power and the latency and bandwidth between various local data centers as the objective function of the genetic algorithm, comprehensively evaluate the computing power of heterogeneous resource devices used in each data center, the latency and bandwidth between data centers, and the number of nodes allocated to each pipeline stage. Through optimization and mutation using the genetic algorithm, the final hybrid parallel structure is obtained, and pipeline parallel groups and data parallel groups are set for nodes in the heterogeneous environment.

[0069] Based on the set pipeline scale and the calculated resources required for each Stage, the optimal combination is obtained using a genetic algorithm. First, an optimization objective function is set, which considers two aspects: 1) Task resource balancing: Since there are different computing devices with different capabilities, such as A100 and T4, the resources obtained in each Stage queue need to be close to stage_task. We choose to use a variance to achieve this. The smaller the variance, the more reasonable the resource allocation.

[0070] computility-Cost=(resource-stage_task) 2 (6)

[0071] Here, Resource represents the actual allocated computing power.

[0072] 2) Cross-domain computation cost: Considering that the nodes in each Stage group may be composed of different data centers, the total cost between nodes in the task group is calculated based on the bandwidth and latency between nodes. The formula is shown in (6). The objective function of the genetic algorithm design is to minimize resource consumption and the additional communication cost caused by distance:

[0073] min(Comm-Cost)=Pipeline_Cost+Data_Cost (7)

[0074] Where Pipeline_Cost represents the cost of pipeline parallel communication, and Data_Cost represents the communication cost generated by data parallelism.

[0075] minCost=λ1(computability-Cost)+λ2(Comm-Cost) (8)

[0076] Wherein, minCost represents the objective function of the genetic algorithm, which is to minimize cost; computation-Cost represents the degree of matching of computing power among pipeline stage groups, i.e., formula (6); and Comm-Cost represents the additional communication cost caused by latency and bandwidth during model training. λ1 and λ2 are heuristic thresholds used, which are 1 and 10 respectively, to make the allocation strategy pay more attention to the latency caused by cross-domain issues.

[0077] It is understandable that the formula for allocating computing power is to better execute computing tasks. Allocating too much computing power will lead to resource waste, while allocating too little will lead to memory overflow.

[0078] The additional communication caused by latency and bandwidth manifests in: 1) Pipeline parallelism: Since different stages are essentially cross-domain, there are additional communication costs for data transfer between stages. 2) Data parallelism: This is because data parallelism within the same stage uses the allreduce method for model synchronization, so if one node is slow, other nodes will wait, thus generating additional communication costs.

[0079] Through continuous iterative selection, crossover, and mutation using genetic algorithms, different allocation schemes are tried, and the objective function value is gradually optimized to find the best task allocation scheme. Finally, the pipeline parallel group and data parallel group corresponding to each node in the heterogeneous environment are determined, thereby reducing the computational cost brought about by resource balancing and cross-domain operations, and improving the training efficiency of large models.

[0080] Step 4: Train the large model according to the determined hybrid parallel group. First, split the large model according to the pipeline scale set by the user, dividing the entire model into multiple Transform-Layers. At the same time, create multiple model replicas for data parallelism.

[0081] Step 5: Divide the training data into training and validation sets. Then, shard the training set for data parallelism to expand the model training scale. Set the batch size for the sharded data. In order to improve the training throughput in the pipeline stage, 1F1B is selected as the pipeline parallelism strategy, and the batch size is further divided into smaller mini-batch sizes.

[0082] Step 6: Input the mini-batch size into the first stage of the pipeline for forward propagation. After forward propagation, proceed to the second stage for further forward propagation, and so on, until the forward propagation is complete. After the forward propagation is complete, calculate the loss between the predicted and actual results using the loss function, and then perform backpropagation on the entire pipeline to calculate the gradient. Since data parallelism is used, we employ an allreduce synchronous update strategy, aggregating the gradient data calculated by each node of the same pipeline stage to update the model parameters. This means that the global stage achieves pipeline parallelism, while each stage achieves data parallelism and model synchronization.

[0083] Figure 2This is a reference diagram of a large-scale hybrid parallel training architecture for heterogeneous environments. Taking the model divided into four stages as an example, consisting of four data centers in different regions, this demonstrates the model training in a heterogeneous environment proposed in this embodiment. The specific research content is divided into the following parts:

[0084] (1) Sorted by computing power scale, A>C>D>B. Among them, location A is the larger data center, and it also has two types of resource equipment; determine the latency bandwidth of each data center; then determine the final pipeline group and data parallel group. It can be seen that the two data centers with the largest latency are used as the first and last nodes, and there is no communication between them.

[0085] Since location A has the largest computing power, node A is logically split, assigning it the tasks of two stages: Stage 1 and Stage 2. Meanwhile, because nodes B and D have smaller computing power, data centers B and D are logically reorganized to share a single Stage 3 task. Location C has a medium computing power and will therefore undertake a single Stage 4 task.

[0086] (2) Two types of computing resources were used to train the large model, including A100 and T4 GPUs, to train a large-scale generative language model based on Transform. The network used was a non-dedicated network with different latency and bandwidth. In order to better utilize the computing power of each device in each place, each input will be "linked" to each stage according to each computing power. Here, the throughput of each device will be statistically analyzed in advance.

[0087] (3) By implementing "many-to-many" links between different device resources, the fixed structure of traditional pipelines and data parallelism is replaced, for example in... Figure 2 After the mini-batch size of Stage 1 in the pipeline is processed by the A100, the output needs to be passed as input to the next pipeline Stage 2 for further processing. At this point, our task allocation strategy can select multiple T4 cards to perform the computation, thus maximizing the computing power of the A100. Simultaneously, if a computing device has significantly more computational capacity, it can process inputs from multiple previous stages and distribute its output across several weaker devices to maximize utilization. Figure 2 In Stage 2, which is a parallel pipeline, A100 corresponds to the preceding and following nodes.

[0088] During forward propagation, each Stage propagates its computation results as input to the next Stage. For backpropagation, each node in each Stage computes gradient information and aggregates the gradient information using the Allreduce method, achieving synchronous updates within its respective Stage group. This synchronous update strategy effectively ensures model convergence.

[0089] Example 2

[0090] The purpose of this embodiment is to provide a large-scale model hybrid parallel training system for heterogeneous environments, including:

[0091] The acquisition module is used to acquire the computing power of different data centers in a heterogeneous environment and sort the network latency between different data centers.

[0092] The allocation module is used to assign the two data centers with the highest network latency to the first and last training stages of the pipeline parallel training of the model to be trained, based on the ranking results of network latency between different data centers.

[0093] Optimization module: Based on the pipeline parallel scale of the model to be trained, with the goals of balanced computing power of data centers between each training stage and equivalence of computing task transmission among nodes in each training stage, iterative optimization is performed using a genetic algorithm to obtain the data center corresponding to each training stage and the computing tasks of different nodes within each training stage.

[0094] Training module: Based on the data center corresponding to each training stage and the computing tasks of different nodes within each training stage, the model to be trained is trained.

[0095] Example 3

[0096] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0097] Example 4

[0098] The purpose of this embodiment is to provide a computer-readable storage medium.

[0099] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0100] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0101] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0102] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A hybrid parallel training method for large models in heterogeneous environments, characterized in that, include: Obtain the computing power of different data centers in a heterogeneous environment and sort the network latency between different data centers; Based on the ranking of network latency among different data centers, the two data centers with the highest network latency are assigned to the first and last training stages of the pipeline parallel training of the model to be trained. Based on the pipeline parallel scale of the model to be trained, with the goals of balanced computing power of the data center between each training stage and equivalence of computing task transmission among the nodes contained in the data center in each training stage, the genetic algorithm is used for iterative optimization to obtain the data center corresponding to each training stage and the computing tasks of different nodes in each training stage. The genetic algorithm is optimized with the objective function being the matching degree of computing power in the pipelined parallel training phase and the additional communication cost caused by latency bandwidth during training. Based on the network latency of the data center nodes, a genetic algorithm is used to group the nodes to obtain the nodes in each training stage; The model to be trained is trained based on the data center corresponding to each training stage and the computational tasks of different nodes within each training stage.

2. The hybrid parallel training method for large models in heterogeneous environments as described in claim 1, characterized in that, In each data center, a node is selected as a representative node. The Floyd-Warshall algorithm is used to calculate the distance between the representative nodes in different data centers. The network latency of different data centers is determined based on the distance calculation results of the representative nodes.

3. The hybrid parallel training method for large models in heterogeneous environments as described in claim 1, characterized in that, The computing resources allocated to each training stage are determined based on the overall computing power of different data centers and the parallel scale of the model to be trained.

4. The hybrid parallel training method for large models in heterogeneous environments as described in claim 1, characterized in that, In each training phase, all-reduce is used as the synchronization strategy, and each node communicates in a point-to-point manner.

5. The hybrid parallel training method for large models in heterogeneous environments as described in claim 1, characterized in that, The training data of the model is processed in a pipelined parallel manner through nodes in different training stages to perform forward and backward computations.

6. A hybrid parallel training system for large models in heterogeneous environments, characterized in that, A hybrid parallel training method for large models in heterogeneous environments as described in any one of claims 1-5 includes: The acquisition module is used to acquire the computing power of different data centers in a heterogeneous environment and sort the network latency between different data centers. The allocation module is used to assign the two data centers with the highest network latency to the first and last training stages of the pipeline parallel training of the model to be trained, based on the ranking results of network latency between different data centers. Optimization module: Based on the pipeline parallel scale of the model to be trained, with the goals of balanced computing power of data centers between each training stage and equivalence of computing task transmission among nodes in each training stage, iterative optimization is performed using a genetic algorithm to obtain the data center corresponding to each training stage and the computing tasks of different nodes in each training stage. Training module: Based on the data center corresponding to each training stage and the computing tasks of different nodes within each training stage, the model to be trained is trained.

7. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform a hybrid parallel training method for large models in heterogeneous environments as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is executed by a processor as described in any one of claims 1 to 5, a hybrid parallel training method for large models in heterogeneous environments.

Images