A globally unified large model pipeline parallel training optimization method and system

Abstract

The application discloses a global unified large model pipeline parallel training optimization method and system, which comprises the following steps: obtaining a large model to be trained and converting it into an operator sequence; constructing a model division sub-model, which takes minimizing the maximum computing load of each pipeline stage on the corresponding training device as the optimization target; constructing a stage deployment sub-model, which takes minimizing the transmission delay between pipeline parallel stages as the optimization target; constructing a strategy scheduling sub-model, which takes minimizing the communication waiting time between pipeline nodes and the bubble time of each scheduling strategy during the execution of each pipeline stage as the optimization target; constructing a global unified joint solution model based on the model division, stage deployment and strategy scheduling sub-models, and jointly solving the model to minimize the total delay of a single training cycle as the global optimization target, so as to obtain a target parallel configuration including a model division scheme, a stage deployment scheme and a strategy scheduling scheme. The application can improve the large model pipeline parallel training efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence model training technology, specifically to a globally unified method and system for optimizing parallel training of large model pipelines. Background Technology

[0002] Pre-trained large-scale language models have a massive number of parameters, and the memory capacity of a single GPU is insufficient to support the training of the entire model. Therefore, parallel and distributed training have become important methods for training large models. Pipeline parallelism, in particular, divides the model into multiple stages and deploys them across multiple GPUs, allowing different micro-batch data to be executed sequentially in a pipelined manner, thereby alleviating the memory pressure on a single device and improving training efficiency. From a system perspective, pipeline parallelism mainly involves three dimensions: model partitioning, stage deployment, and policy scheduling. Model partitioning determines the computational load of each pipeline stage and the communication overhead between stages; stage deployment determines the mapping relationship between pipeline stages and physical devices; and policy scheduling determines the execution order of micro-batch training tasks between stages and the resulting bubble time.

[0003] In existing technologies, HetPipe, AutoPipe-H, CAPSlog, and HelixPip primarily optimize model partitioning; Alpa, SDPipe, and HPipe focus on stage deployment or local joint optimization of model partitioning and deployment; while GPipe, PipeDream, interleaved pipelines, bidirectional pipelines, Tessel, and Koala mainly design or search pipeline scheduling strategies. While these methods can improve load balancing, reduce communication latency, or minimize pipeline bubbles from a single dimension, there is a clear coupling between model partitioning, stage deployment, and strategy scheduling: model partitioning affects stage computational load, stage deployment affects cross-device communication latency, and both further affect bubble time and end-to-end training efficiency under the scheduling strategy. Therefore, optimizing only one dimension independently is insufficient to achieve globally optimal performance for pipelined parallel training. Furthermore, current large-scale model training often still relies on researchers manually configuring model partitioning, stage deployment, and scheduling strategies based on experience, and repeatedly experimenting on real GPU clusters, resulting in high time costs, high GPU resource consumption, and high economic costs.

[0004] Therefore, there is an urgent need for a large-scale model pipeline parallel training optimization method that can comprehensively consider model partitioning, stage deployment, and strategy scheduling from a global and unified perspective, and jointly optimize the total latency of a single training cycle. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a globally unified method and system for optimizing parallel training of large model pipelines, which integrates model partitioning, stage deployment and strategy scheduling into the same optimization framework and solves them jointly, automatically generates globally adapted target parallel configurations, reduces manual trial and error costs, reduces the total latency of a single training cycle, and improves the end-to-end execution efficiency of parallel training of large model pipelines.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A globally unified method for optimizing parallel training of large-scale model pipelines includes: Obtain the large model to be trained and convert it into a sequence of operators consisting of multiple operators; A model partitioning sub-model is constructed, which is used to divide the operator sequence into multiple consecutive pipeline stages, with the optimization objective being to minimize the maximum computational load of each pipeline stage on the corresponding training device. A phase deployment sub-model is constructed, which is used to map each pipeline stage to the physical training device in the training cluster, with the optimization objective of minimizing the transmission latency between parallel pipeline stages. A strategy scheduling sub-model is constructed, which is used to sort the multiple micro-batch training tasks formed by dividing the training batches among each pipeline stage, and the optimization objective is to minimize the communication waiting time between pipeline nodes during the execution of each pipeline stage and the bubble time during the execution of each scheduling strategy. Based on the model partitioning sub-model, stage deployment sub-model and strategy scheduling sub-model, a globally unified joint solution model is constructed. With minimizing the total latency of a single training cycle as the global optimization objective, the globally unified joint solution model is jointly solved to obtain the target parallel configuration including the model partitioning scheme, stage deployment scheme and strategy scheduling scheme. The total delay is characterized by the time overhead corresponding to the optimization objectives of the model division sub-model, the phased deployment sub-model, and the strategy scheduling sub-model.

[0007] Furthermore, the objective function corresponding to the optimization objective of the model partitioning sub-models. The expression is as follows:

[0008] In the above formula, This represents the s pipeline stages obtained by dividing the operator sequence; s represents the number of pipeline stages; j represents the pipeline stage number; p and q represent the starting operator number and ending operator number corresponding to the current pipeline stage, respectively; i represents the operator number. Operator arrive The subsequence in the th The computational overhead of each pipeline stage.

[0009] Furthermore, the constraints corresponding to the optimization objective of the model partitioning sub-model include one or more of the following: device memory constraints, operator unique allocation constraints, operator coverage and mutual exclusion constraints, and continuous partitioning constraints. The device memory constraint is used to limit the total memory overhead of each pipeline stage on the corresponding training device to no more than the total memory capacity of the corresponding training device, as expressed below:

[0010] In the above formula, This indicates the memory overhead of the model parameters. This indicates the gradient memory overhead. This represents the memory overhead of operator o, and p and q represent the start operator index and end operator index corresponding to the current pipeline stage, respectively. This represents the video memory capacity of the r-th training device; The operator unique assignment constraint is used to limit each operator to be assigned to only one training device, and the expression is as follows:

[0011] In the above formula, Let represent the o-th operator, where o represents the operator index, r represents the training device index, and d represents the number of training devices. Operator Deployed on the r-th training device, This indicates that there exists one and only one training device that satisfies the condition; The operator coverage and mutual exclusion constraints are used to limit the number of pipeline stages to cover all operators in the operator sequence and to ensure that there are no duplicate operators between different pipeline stages. The expression is as follows:

[0012] In the above formula, This represents the k-th pipeline stage, where k is the pipeline stage number and s is the number of pipeline stages. N represents all operators in the operator sequence, and N represents the total number of operators. and These represent different stages of the production line. Let s be the union of s pipeline stages. This indicates that there are no overlapping operators between different pipeline stages; The continuous partitioning constraint is used to limit each pipeline stage to consist of operators that are consecutively positioned in the operator sequence, as expressed below:

[0013] In the above formula, and These represent the start operator number and end operator number corresponding to the k-th pipeline stage, respectively. This represents the operators arranged sequentially in the operator sequence within the k-th pipeline stage.

[0014] Furthermore, the model partitioning sub-model employs a dynamic programming algorithm. This algorithm enumerates the previous partition position and the currently selected training device, compares the optimized computational load of the preceding pipeline stage with the computational load of the current continuous operator sub-sequence on the selected training device, and takes the larger of the two as the maximum stage load in the current partition state. Then, it selects the scheme that minimizes this maximum stage load from all available partition positions and training devices as the final scheme. The expression for the dynamic programming algorithm is as follows:

[0015] In the above formula, Indicates the current processing to the operator. And operators When deployed to device r, the minimum and maximum computational costs obtained in a heterogeneous cluster where the device usage status is masked. o represents the operator sequence number, r represents the device sequence number, mask represents the device usage mask, D represents the set of training devices in the cluster, u represents the currently deployed device selected from the set D, and k represents the partition position in the operator sequence. This represents the minimum and maximum computational cost when the first k operators are deployed to the first r-1 devices after removing device u. This represents the computational cost of deploying the (k+1)th to the 0th operators onto device u.

[0016] Furthermore, the objective function corresponding to the optimization objective of the stage deployment sub-model is... The expression is as follows:

[0017] In the above formula, Indicates the assembly line stage The mapping relationship between physical training equipment, Let s represent the i-th pipeline stage, and s represent the number of pipeline stages. and Let represent the computation time of the i-th and j-th pipeline stages, respectively. and These represent the communication times related to the i-th and j-th pipeline stages, respectively, and b represents the number of micro-batches. This represents the bubble time generated by s production line stages; The constraints corresponding to the optimization objective of the stage deployment sub-model include memory capacity constraints and stage unique deployment constraints. The memory capacity constraints limit the total memory overhead of pipeline stages deployed to the same training device to not exceed the memory capacity of the training device. The stage unique deployment constraints limit each pipeline stage to be deployed to only one training device.

[0018] Furthermore, the objective function corresponding to the optimization objective of the policy scheduling sub-model is... The expression is as follows:

[0019] In the above formula, Q represents the candidate scheduling strategy, Y represents the set of candidate scheduling strategies, s represents the number of pipeline stages, and j represents the pipeline stage number. This represents the j-th pipeline stage. and This indicates the start and end times of the j-th pipeline stage. This represents the effective computation time of the o-th training task or operator in the j-th pipeline stage. It represents the sum of the effective computation time of all training tasks or operators within the j-th pipeline stage; The constraints corresponding to the optimization objective of the policy scheduling sub-model include task dependency constraints, which limit the execution order of each micro-batch training task to ensure that any task can only be scheduled and executed after its corresponding parent task has been completed.

[0020] Furthermore, the total delay in the optimization objective of the globally unified joint solution model includes one or more of the following: the maximum computational load of each pipeline stage in the model partitioning optimization model, the cross-device communication delay between pipeline parallel stages in the stage deployment optimization model, and the total bubble time of the scheduling strategy in the strategy scheduling optimization model. The corresponding objective function expression is as follows:

[0021] In the above formula, Latency represents the total latency of a single training cycle. Operator The division results This represents the o-th operator. Indicates the assembly line stage The results of the equipment deployment Let Q represent the j-th pipeline stage, Q represent the candidate scheduling strategy, s represent the number of pipeline stages, and b represent the number of micro-batches. This represents the stage computation cost of the j-th pipeline stage. This represents the cross-device communication latency between the j-th pipeline stage and the (j+1)-th pipeline stage. This represents the total bubble time corresponding to scheduling strategy Q.

[0022] A globally unified large model pipeline parallel training optimization system includes interconnected microprocessors and memory, wherein the microprocessors are programmed or configured to execute the steps of a globally unified large model pipeline parallel training optimization method.

[0023] A computer-readable storage medium storing a computer program for being programmed or configured by a microprocessor to perform steps of a globally unified large model pipeline parallel training optimization method.

[0024] A computer program product includes a computer program / instructions that are programmed or configured to execute steps of a globally uniform large model pipeline parallel training optimization method via a processor.

[0025] Compared with the prior art, the advantages of the present invention are as follows: This invention models the three key aspects of parallel training in large-scale pipelined systems: model partitioning, stage deployment, and policy scheduling. It further constructs a globally unified joint solution model, enabling the computational load from model partitioning, the cross-device transmission latency from stage deployment, and the communication wait time and bubble time from policy scheduling to be considered jointly within the same optimization framework. This avoids the problem of limited overall training efficiency caused by local optimization of only a single aspect in existing technologies. By using the minimum total latency of a single training cycle as the global optimization objective for joint solution, the model partitioning scheme, stage deployment scheme, and policy scheduling scheme can be adapted to each other, thereby reducing load imbalance between pipeline stages, reducing cross-device communication wait time and pipeline bubble time, and improving the utilization of training equipment and the end-to-end execution efficiency of pipelined parallel training. By automatically generating target parallel configurations, it reduces the need for researchers to rely on experience for manual configuration and repeated trial and error in real GPU clusters, lowering the time and computational resource costs of designing large-scale pipelined parallel training solutions. Attached Figure Description

[0026] Figure 1 This is a flowchart of a globally unified large-model pipeline parallel training optimization method according to an embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram of the total delay in pipeline parallelism in a specific application embodiment.

[0028] Figure 3 This is a schematic diagram of the pipeline parallel simulation process interface in a specific application embodiment.

[0029] Figure 4 This is a schematic diagram of the visualization interface for pipeline parallel simulation in a specific application embodiment. Detailed Implementation

[0030] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0031] The following is a definition of some of the technical terms used in this invention: Large models: Neural network models are a frequently mentioned concept in deep learning, which is built upon neural networks. Large models refer to neural network models with a certain number of parameters. These models share common characteristics: a large number of parameters, long training time, and high training cost. They cannot be trained on a single GPU and must be trained using distributed training methods.

[0032] Pipeline point-to-point communication: In pipeline parallel training scenarios, computing devices (such as GPUs) that deploy different model stages transmit data through direct connections (without the need for intermediate nodes to forward data). Its core is to support the flow of micro-batch data between different stages of the pipeline, and it is a key communication form that affects the efficiency of pipeline training.

[0033] Pipeline bubbles: During pipeline parallel training, the "idle time" generated when computing devices (such as GPUs) cannot continuously execute computing tasks due to data dependencies, communication delays, or load imbalances is one of the core bottlenecks affecting pipeline training efficiency, and it runs through the optimization logic of three dimensions: model partitioning, stage placement, and scheduling strategies.

[0034] Pipeline load balancing refers to using specific methods to distribute the computational costs, memory consumption, and communication overhead of each stage in pipeline parallel training as evenly as possible.

[0035] Pipeline parallelism: Pipeline parallelism is a common parallel and distributed training method in large model training. Pipeline parallelism divides the model into multiple stages and places them on different GPUs, passing activations between stages along the model propagation direction. During this process, training batches are further divided into multiple micro-batches, which are passed between stages in a pipeline manner, forming a time-overlapping computation pattern. This achieves memory optimization and computational parallelism, thereby realizing efficient utilization of computing resources and accelerating the training process.

[0036] Model partitioning: Model partitioning is the first dimension of pipelined parallelism and its foundation. Its goal is to divide a large language model into several consecutive pipeline stages based on its layer structure or parameters, thereby adapting the memory consumption of each stage to the memory capacity of a single computing device.

[0037] Stage Deployment: Stage deployment is the second dimension of pipeline parallelism, referring to the process of assigning divided pipeline stages to specific computing devices. This process aims to adapt to hardware topologies that encompass intra-node NVLink, inter-node Ethernet, and InfiniBand, thereby minimizing inter-stage communication latency.

[0038] Policy scheduling: Policy scheduling is the core manifestation of pipelined parallelism and the core mechanism for controlling data flow and computation timing at each stage of the pipeline. The scheduling policy determines the performance of pipelined parallelism, including the degree of overlap between communication and computation, bubble rate, and overall training throughput.

[0039] Basic pipelined parallel systems refer to GPipe, PipeDream, interleaved pipelines, and bidirectional pipelines.

[0040] Complex pipelined parallel systems: pipelined parallelism formed by combining two or more methods in a basic pipelined parallel system.

[0041] A globally unified pipelined parallel simulation system: This system comprehensively considers the three dimensions of pipelined parallelism (model partitioning, stage placement, and strategy scheduling), covers all basic pipelined parallelism methods in current research, and achieves end-to-end pipelined parallel training efficiency simulation.

[0042] The massive number of parameters in pre-trained Large Language Models (LLMs) has led to memory consumption exceeding the current GPU memory capacity limit. To train large models effectively and efficiently, parallel and distributed computing have become core methods. Among these, pipelined parallelism, with its partitioning of massive model parameters, not only alleviates the memory pressure on a single GPU but also allows for training data across multiple GPUs in a pipelined manner, thereby improving model training efficiency and playing an irreplaceable role in large model training.

[0043] From a system perspective, pipelined parallelism can generally be divided into three dimensions: model partitioning, stage deployment, and policy scheduling. Model partitioning is the foundation of pipelined parallelism, affecting the computational load and communication overhead between different stages of the pipeline. Stage deployment, based on model partitioning, maps pipeline stages to physical computing devices. Policy scheduling is the core mechanism of pipelined parallelism, optimizing pipelined parallel training performance by ordering micro-batches among pipeline stages.

[0044] In terms of model partitioning, existing technologies mainly optimize pipeline stage partitioning and load balancing. For example, HetPipe integrates pipeline parallelism and data parallelism, aggregating heterogeneous GPUs through virtual worker nodes to support large-scale model training. Its partitioning logic minimizes the maximum execution time between stages (including communication and computation time) and meets memory requirements with the help of the CPLEX optimizer. AutoPipe-H enhances the flexibility of partitioning Transformer models by dividing Transformer blocks into residual attention blocks and residual feedforward network blocks, expanding the partitioning search space and improving load balancing potential without increasing communication overhead. At the same time, for the complex dependencies between the one-forward-one-backward (1F1B) and cooling stages in pipeline execution, AutoPipe-H renumbers the stages to accurately simulate iteration time and uses dynamic programming to determine key stages on the critical path to achieve fast search for the optimal partitioning. CAPSlog optimizes the workflow partitioning by removing irrelevant layers during performance analysis, retaining only the layer to be analyzed and necessary upstream and downstream virtual layers. Candidate partitions are generated through induction, and binary search is used to determine the lowest peak memory usage. HelixPip, on the other hand, divides the Transformer layer into parameterized pre-attention modules, parameterless attention modules, and parameterized post-attention modules. It employs a spiral mapping strategy to remove attention computation from the pipeline bubble, making the pipeline bubble only related to the execution time of the pre-attention and post-attention modules.

[0045] In terms of stage deployment, existing technologies mainly focus on optimizing the mapping between pipeline stages and physical devices. For example, Alpa focuses on automating 3D parallelism in large-scale language model training, decoupling 3D parallelism into intra-operator parallelism and inter-operator parallelism, where inter-operator parallelism is another manifestation of pipeline parallelism. In the stage placement task, Alpa uses device mesh topology and latency metrics for each stage as input, and solves the stage placement problem through dynamic programming to find the stage-device mapping and minimize end-to-end latency in pipeline parallelism. SDPipe proposes a semi-decentralized framework for optimizing stage placement in training clusters, splitting synchronization operations into centralized communication topology and decentralized peer-to-peer communication, thereby achieving a balance between high heterogeneity tolerance and convergence efficiency. HPipe addresses the stage placement problem in training clusters by jointly optimizing model partitioning and stage placement. First, it uses dynamic programming to determine the model hierarchical scheme and sequence splitting scheme, and then pipelines the subsequences across heterogeneous devices based on the token dimension at runtime.

[0046] In terms of policy scheduling, representative existing technologies include GPipe, PipeDream, interleaved pipelines, bidirectional pipelines, and automated pipeline scheduling policy generation methods such as Tessel and Koala. GPipe first divides the model into consecutive stages, assigning each stage to a single GPU. During training, a batch is split into multiple micro-batches, which are then processed through a cross-device pipeline. This design achieves basic computational overlap, but since micro-batches must be processed sequentially, pipeline bubbles are still unavoidable. PipeDream proposes an asynchronous execution mechanism for forward and backward propagation, caching activation values ​​and gradients within the pipeline, allowing downstream stages to start computation without waiting for upstream to be fully synchronized, and proposes a 1F1B scheduling strategy. Interleaved pipelines optimize resource utilization, reduce idle time, and improve hardware utilization by supporting the simultaneous execution of multiple model partitions on a single device. Chimera uses bidirectional pipeline technology to reduce pipeline bubbles and load unevenness, but each card needs to store copies of sub-models in two directions, resulting in double the GPU memory overhead. Zero-bubble pipelines aim to eliminate device idle time through precise task scheduling and dynamic micro-batch partitioning, and achieve cross-GPU task overlap execution by rationally arranging forward and backward propagation tasks. Tessel reduces the search space by leveraging repetition patterns common in efficient pipeline scheduling, decomposing the scheduling search problem into two stages: repetition unit construction and schedule completion, and using the Z3 solver to determine the optimal repetition unit. Koala, based on fundamental dependencies applicable to existing pipeline scheduling strategies, formalizes the pipeline scheduling problem as a binary tree traversal problem, and uses the Try-Test genetic algorithm to automatically search for the optimal strategy within the structured scheduling strategy search space.

[0047] However, it's easy to see that model partitioning, stage deployment, and policy scheduling are not independent of each other. Model partitioning and stage deployment determine the computational load and communication overhead at each stage of pipelined parallelism. These computational loads and communication overheads, in turn, affect task waiting time and bubble time during policy scheduling, thus determining the bubble rate under different scheduling policies and ultimately impacting pipelined parallel training efficiency. Most existing pipelined parallel training methods only optimize one dimension of model partitioning, stage deployment, or policy scheduling. While this can improve pipelined parallel performance to some extent, the performance improvement remains limited due to the lack of unified modeling and joint solution of the coupling relationships among model partitioning, stage deployment, and policy scheduling.

[0048] Furthermore, in large-scale pipelined parallel training, achieving optimal training efficiency typically requires researchers to rely on extensive prior knowledge, manually configure model partitioning methods, stage deployment schemes, and policy scheduling methods, and continuously experiment with different pipelined parallel strategies within the training cluster. This process is not only time-consuming but also consumes significant GPU resources, resulting in high economic costs. Therefore, current technologies still lack a solution that can comprehensively consider model partitioning, stage deployment, and policy scheduling from a global, unified perspective, and obtain optimized end-to-end pipelined parallel training configurations through a unified solution method.

[0049] Therefore, based on the shortcomings of the existing technology, this invention provides a globally unified optimization method for large model pipeline parallel training. It incorporates model partitioning, stage deployment, and strategy scheduling into the same solution framework, and jointly determines the model partitioning scheme, stage deployment scheme, and strategy scheduling scheme under a unified objective. This reduces the time cost and GPU resource overhead required for manually designing pipeline parallel strategies, and improves the overall efficiency of large model pipeline parallel training.

[0050] like Figure 1 As shown, this embodiment of the invention provides a globally unified method for optimizing parallel training of large model pipelines, including: Obtain the large model to be trained and convert it into a sequence of operators consisting of multiple operators; A model partitioning sub-model is constructed to divide the operator sequence into multiple consecutive pipeline stages, with the optimization objective being to minimize the maximum computational load of each pipeline stage on the corresponding training device. A phased deployment sub-model is constructed to map each pipeline stage to the physical training device in the training cluster, with the optimization objective of minimizing the transmission latency between parallel pipeline stages. A strategy scheduling sub-model is constructed to sort the multiple micro-batch training tasks formed by dividing the training batches among the pipeline stages, with the optimization objective being to minimize the communication waiting time between pipeline nodes and the bubble time during the execution of each scheduling strategy. A globally unified joint solution model is constructed based on model partitioning sub-model, stage deployment sub-model and strategy scheduling sub-model. The global optimization objective is to minimize the total latency of a single training cycle. The globally unified joint solution model is jointly solved to obtain the target parallel configuration including model partitioning scheme, stage deployment scheme and strategy scheduling scheme. The total latency is characterized by the time overhead corresponding to the optimization objectives of the model partitioning sub-model, the phased deployment sub-model, and the strategy scheduling sub-model.

[0051] In this embodiment, model partitioning in pipeline parallelism aims to achieve load balancing between pipeline stages, thereby reducing bubbles in policy scheduling implementation during pipeline parallelism. For the input model... It can be converted into a length of operator sequence The corresponding memory overhead is If the sequence is divided into Each stage of the production line, assuming Operator In the The computational overhead at each stage Operator arrive The subsequence in the th The computational overhead at each stage. With the goal of achieving load balancing across the training cluster, the model partitioning problem is formalized as a constrained minimax optimization problem: assuming the operator sequence is divided into s subsequences, the objective is to minimize the maximum computational load of each GPU (as shown in equation (1)), i.e., to minimize the load differences between heterogeneous devices through reasonable partitioning:

[0052] In the above formula, This represents the s pipeline stages obtained by dividing the operator sequence; s represents the number of pipeline stages; j represents the pipeline stage number; p and q represent the starting operator number and ending operator number corresponding to the current pipeline stage, respectively; i represents the operator number. Operator arrive The subsequence in the th Computational overhead at each stage of the pipeline; This indicates the memory overhead of the model parameters. This indicates the gradient memory overhead. This represents the memory overhead of operator o, and p and q represent the start operator index and end operator index corresponding to the current pipeline stage, respectively. This represents the video memory capacity of the r-th training device; Let represent the o-th operator, where o represents the operator index, r represents the training device index, and d represents the number of training devices. Operator Deployed on the r-th training device, This indicates that there exists one and only one training device that satisfies the condition; This represents the k-th pipeline stage, where k is the pipeline stage number and s is the number of pipeline stages. Let N represent all operators in the operator sequence, and let N represent the total number of operators. and These represent different stages of the production line. Let s be the union of s pipeline stages. This indicates that there are no overlapping operators between different pipeline stages; and These represent the start operator number and end operator number corresponding to the k-th pipeline stage, respectively. This represents the operators arranged sequentially in the operator sequence within the k-th pipeline stage.

[0053] It is understandable that the feasibility of the solution is ensured by constructing four major constraints: Formula (1a) represents the device memory constraint, the total memory overhead of the pipeline stage must not exceed the total memory of the device, thus ensuring that the solution is effective in the training cluster; Formula (1b)'s operator unique allocation constraint ensures that each operator is allocated to one and only one device; Formula (1c)'s operator coverage and mutual exclusion constraint requires that each operator be computed and only once; Formula (1d)'s continuous partitioning constraint requires that each subsequence consists of consecutive operators. Model partitioning can reduce the load imbalance between different pipeline stages during the model partitioning stage, avoid some training devices becoming computational bottlenecks, and provide a more balanced computational load for subsequent stage deployment and strategy scheduling.

[0054] For the minimax optimization objective of formula (1), a comparative analysis of the characteristics of mainstream solution algorithms is conducted: the greedy algorithm is efficient but prone to getting trapped in local optima; the integer linear programming algorithm can guarantee the global optimum but has a time complexity of O(2^3). n However, traditional methods like `std::runtime` are difficult to adapt to large models; while dynamic programming algorithms can obtain the global optimum in O(n²) time complexity and can naturally integrate into the hardware constraints of heterogeneous GPUs. Therefore, dynamic programming is used to derive the optimal partitioning scheme with the goal of "minimizing the maximum load on each GPU". In this embodiment, the model partitioning into sub-models employs a dynamic programming algorithm. The dynamic programming algorithm enumerates the previous partition position and the currently selected training device, compares the optimized computational load of the preceding pipeline stage with the computational load of the current continuous operator sub-sequence on the selected training device, and takes the larger of the two as the maximum stage load in the current partition state. Then, it selects the scheme that minimizes this maximum stage load from all available partition positions and training devices as the final scheme. The dynamic programming algorithm expression is as follows: (2) In the above formula, Indicates the current processing to the operator. And operators When deployed to device r, the minimum and maximum computational costs obtained in a heterogeneous cluster where the device usage status is mask, where o represents the operator index, r represents the device index, and mask represents the device usage mask (which is a... A binary number with 1 bit, if the 1st bit... A bit value of 1 indicates the device (In use), D represents the set of cluster training devices, u represents the currently deployed device selected from the device set D, and k represents the partition position in the operator sequence. This represents the minimum and maximum computational cost when the first k operators are deployed to the first r-1 devices after removing device u. This represents the computational cost of deploying the (k+1)th to the 0th operators onto device u. It is the optimization goal.

[0055] Staged deployment involves dividing the model into pipeline stages and deploying them on the training cluster. This maps the stages to physical devices, minimizing latency between parallel pipeline stages. In this embodiment, the objective function corresponding to the optimization goal of the staged deployment sub-model is... The expression is as follows:

[0056] In the above formula, Indicates the assembly line stage The mapping relationship between physical training equipment, Let s represent the i-th pipeline stage, and s represent the number of pipeline stages. and Let represent the computation time of the i-th and j-th pipeline stages, respectively. and These represent the communication times related to the i-th and j-th pipeline stages, respectively, and b represents the number of micro-batches. This represents the bubble time generated by s production line stages.

[0057] The constraints corresponding to the optimization objective of the stage deployment sub-model include memory capacity constraints (Formula 3a) and stage unique deployment constraints (Formula 3b). The memory capacity constraints limit the total memory overhead of pipeline stages deployed to the same training device to not exceed the memory capacity of the training device. The stage unique deployment constraints limit each pipeline stage to be deployed to only one training device.

[0058] It is understandable that the stage deployment sub-model can combine the communication differences between different devices in the training cluster to reduce the data transmission overhead between adjacent pipeline stages, avoid excessive cross-device communication latency due to unreasonable stage placement, and thus improve the data flow efficiency during pipeline parallel execution.

[0059] Policy scheduling is a core mechanism in pipelined parallelism. It aims to reduce communication latency between pipeline nodes and policy bubbles by sorting the partitioned data across pipeline stages. In this embodiment, the objective function corresponding to the optimization objective of the policy scheduling sub-model is... The expression is as follows: (4) (4a) In the above formula, Q represents the candidate scheduling strategy, Y represents the set of candidate scheduling strategies, s represents the number of pipeline stages, and j represents the pipeline stage number. This represents the j-th pipeline stage. and This indicates the start and end times of the j-th pipeline stage. This represents the effective computation time of the i-th training task or operator in the o-th pipeline stage. It represents the sum of the effective computation time of all training tasks or operators within the j-th pipeline stage.

[0060] The constraints corresponding to the optimization objective of the policy scheduling sub-model include task dependency constraints (Formula 4a), which limit the execution order of each micro-batch training task to ensure that any task can only be scheduled for execution after its corresponding parent task has been completed.

[0061] It is understandable that the policy scheduling sub-model is used to reduce idle waiting caused by task dependencies, forward and backward propagation order and stage load differences during pipeline execution, so that each pipeline stage can maintain continuous working state as much as possible, improve the utilization of training equipment and reduce the additional latency introduced by the scheduling policy.

[0062] In this embodiment, the globally unified joint solution model aims to minimize the total latency of a single training cycle. The total latency includes the maximum computational load of each pipeline stage in the model partitioning optimization model (the computational cost of each stage of model partitioning). Cross-device communication latency between parallel pipeline stages in the staged deployment optimization model And the total bubble time of the scheduling strategy in the strategy scheduling optimization model One or more of the following, the corresponding objective function expressions are as follows:

[0063] In the above formula, Latency represents the total latency of a single training cycle. Operator The division results This represents the o-th operator. Indicates the assembly line stage The results of the equipment deployment Let Q represent the j-th pipeline stage, Q represent the candidate scheduling strategy, s represent the number of pipeline stages, and b represent the number of micro-batches. This represents the stage computation cost of the j-th pipeline stage. This represents the cross-device communication latency between the j-th pipeline stage and the (j+1)-th pipeline stage. This represents the total bubble time corresponding to scheduling strategy Q.

[0064] It is understandable that the globally unified joint solution model considers the basic computation in model partitioning, the pipeline point-to-point communication latency in stage placement, and the bubble latency in scheduling strategy together. It evaluates the total latency of each scheme in a unified optimization space, which can avoid the problem of limited overall training efficiency caused by only optimizing the model partitioning, stage deployment, or strategy scheduling locally. It enables the model partitioning results, stage deployment results, and scheduling strategy to be mutually compatible, reducing the time cost and GPU resource overhead required for manually configuring pipeline parallel strategies through repeated trial and error, and improving the end-to-end execution efficiency of large model pipeline parallel training.

[0065] In one specific implementation, to facilitate the use of the pipelined parallel simulation system proposed in this invention by community researchers, the system can be published to the community as a website. The website targets computer science researchers, system architects, parallel computing engineers, university faculty and students, and other relevant users, providing technical demonstrations, functional introductions, and academic resources of the pipelined parallel simulation system, enabling users to understand and use the system for parallel computing research and application development.

[0066] The system website includes displays of system functional modules, introductions to domain-specific languages, showcases of the latest research results and papers, and an interactive pipelined parallel simulation demonstration. The system functional module displays include end-to-end pipeline simulation, model partitioning scheme generation, stage deployment scheme generation, and strategy scheduling scheme generation. The interactive pipelined parallel simulation demonstration receives user-adjusted simulation parameters and displays the corresponding simulation effects in real time, allowing users to intuitively observe the impact of different pipeline parallel configurations on training efficiency.

[0067] In terms of website interface design, the system website adopts a fixed navigation bar on the left and a content card layout on the right. The left navigation bar displays function entrances such as the homepage, pipeline simulation, model partitioning and generation, phased deployment and model generation, and strategy scheduling scheme generation; the right content area displays system introduction, functional modules, simulation parameter control, simulation result visualization, and the latest developments in the field. The website page can be configured with smooth transition animations and interactive simulation demonstration controls. The overall color scheme is mainly technical blue, and the system functions are displayed through charts, flowcharts, and card explanations.

[0068] like Figure 2As shown, the system website includes a pipeline parallel total latency display interface. This interface is used to display the execution process of multiple micro-batches in different pipeline stages during pipeline parallel training. Task blocks at different positions represent the forward propagation, backward propagation, or waiting state of each micro-batch in the time dimension, thus intuitively demonstrating the bubble time generated during pipeline filling, stable execution, and emptying, and the impact of bubble time on the total latency of a single training cycle. In the figure, multiple numbered squares represent the execution order of different micro-batch tasks in each pipeline stage, and the blank areas between different squares represent the idle waiting time of the pipeline stage.

[0069] like Figure 3 As shown, the system website includes a pipelined parallel simulation workflow interface. The simulation parameter control area is located in the center of this interface page, where users can adjust parameters such as the number of pipeline stages, data parallelism, and processing latency. The simulation process can be triggered by the "Start Simulation" button, or the default parameters can be restored by the "Reset" button. This allows users to intuitively compare the execution performance of the pipelined parallel system under different parameter configurations.

[0070] like Figure 4 As shown, the system website includes a pipeline parallel simulation visualization interface. This interface displays simulation execution results, showing multiple pipeline stages and the data flow relationships between them. It also reflects the execution performance under the current simulation configuration through indicator cards such as throughput, average latency, and resource utilization. The interface also allows users to set up historical records or strategy selection entries, enabling them to compare simulation results under different pipeline parallel strategies or parameter configurations.

[0071] Through the above website design, this invention can not only provide simulation and display functions for model partitioning, stage deployment and strategy scheduling, but also lower the threshold for users to understand and use pipelined parallel simulation systems with interactive pages. This allows researchers to perform end-to-end simulation, performance observation and scheme comparison of different pipelined parallel configurations without repeatedly occupying real GPU clusters for trial and error, thereby reducing the time and economic costs of pipelined parallel system design and implementation.

[0072] The present invention further provides a globally unified large model pipeline parallel training optimization system, including interconnected microprocessors and memory, wherein the microprocessors are programmed or configured to execute the steps of a globally unified large model pipeline parallel training optimization method.

[0073] The present invention further provides a computer-readable storage medium storing a computer program, the computer program being programmed or configured by a microprocessor to execute steps of a globally unified large model pipeline parallel training optimization method.

[0074] The present invention further provides a computer program product, including a computer program / instruction, which is programmed or configured to execute steps of a globally unified large model pipeline parallel training optimization method via a processor.

[0075] The system, medium, and product of the present invention, corresponding to the methods described above, also have the advantages described above.

[0076] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. Computer-readable media include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. The memory is used to store computer programs and / or modules. The processor implements various functions by running or executing the computer programs and / or modules stored in the memory, and by calling data stored in the memory. The memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0077] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A globally unified method for optimizing parallel training of large-scale models in a pipeline, characterized in that, include: Obtain the large model to be trained and convert it into a sequence of operators consisting of multiple operators; A model partitioning sub-model is constructed, which is used to divide the operator sequence into multiple consecutive pipeline stages, with the optimization objective being to minimize the maximum computational load of each pipeline stage on the corresponding training device. A phase deployment sub-model is constructed, which is used to map each pipeline stage to the physical training device in the training cluster, with the optimization objective of minimizing the transmission latency between parallel pipeline stages. A strategy scheduling sub-model is constructed, which is used to sort the multiple micro-batch training tasks formed by dividing the training batches among each pipeline stage, and the optimization objective is to minimize the communication waiting time between pipeline nodes during the execution of each pipeline stage and the bubble time during the execution of each scheduling strategy. Based on the model partitioning sub-model, stage deployment sub-model and strategy scheduling sub-model, a globally unified joint solution model is constructed. With minimizing the total latency of a single training cycle as the global optimization objective, the globally unified joint solution model is jointly solved to obtain the target parallel configuration including the model partitioning scheme, stage deployment scheme and strategy scheduling scheme. The total delay is characterized by the time overhead corresponding to the optimization objectives of the model division sub-model, the phased deployment sub-model, and the strategy scheduling sub-model.

2. The globally unified large-model pipeline parallel training optimization method according to claim 1, characterized in that, The objective function corresponding to the optimization objective of the model sub-model is... The expression is as follows: In the above formula, This represents the s pipeline stages obtained by dividing the operator sequence; s represents the number of pipeline stages; j represents the pipeline stage number; p and q represent the starting operator number and ending operator number corresponding to the current pipeline stage, respectively; i represents the operator number. Operator arrive The subsequence in the th The computational overhead of each pipeline stage.

3. The globally unified large-model pipeline parallel training optimization method according to claim 1, characterized in that, The constraints corresponding to the optimization objective of the model partitioning sub-model include one or more of the following: device memory constraints, operator unique allocation constraints, operator coverage and mutual exclusion constraints, and continuous partitioning constraints. The device memory constraint is used to limit the total memory overhead of each pipeline stage on the corresponding training device to no more than the total memory capacity of the corresponding training device, as expressed below: In the above formula, This indicates the memory overhead of the model parameters. This indicates the gradient memory overhead. Let represent the memory overhead of operator 'o', and p and q represent the start operator index and end operator index corresponding to the current pipeline stage, respectively. This represents the video memory capacity of the r-th training device; The operator unique assignment constraint is used to limit each operator to be assigned to only one training device, and the expression is as follows: In the above formula, Let represent the o-th operator, where o represents the operator index, r represents the training device index, and d represents the number of training devices. Operator Deployed on the r-th training device, This indicates that there exists one and only one training device that satisfies the condition; The operator coverage and mutual exclusion constraints are used to limit the number of pipeline stages to cover all operators in the operator sequence and to ensure that there are no duplicate operators between different pipeline stages. The expression is as follows: In the above formula, This represents the k-th pipeline stage, where k is the pipeline stage number and s is the number of pipeline stages. N represents all operators in the operator sequence, and N represents the total number of operators. and These represent different stages of the production line. Let s be the union of s pipeline stages. This indicates that there are no overlapping operators between different pipeline stages; The continuous partitioning constraint is used to limit each pipeline stage to consist of operators that are consecutively positioned in the operator sequence, as expressed below: In the above formula, and These represent the start operator number and end operator number corresponding to the k-th pipeline stage, respectively. This represents the operators arranged sequentially in the operator sequence within the k-th pipeline stage.

4. The globally unified large-model pipeline parallel training optimization method according to claim 1, characterized in that, The model partitioning process employs a dynamic programming algorithm. This algorithm enumerates the previous partition position and the currently selected training device, compares the optimized computational load of the preceding pipeline stage with the computational load of the current continuous operator subsequence on the selected training device, and takes the larger of the two as the maximum stage load in the current partition state. Then, it selects the solution that minimizes this maximum stage load from all available partition positions and training devices as the final solution. The expression for the dynamic programming algorithm is as follows: In the above formula, Indicates the current processing to the operator. And operators When deployed to device r, the minimum and maximum computational costs obtained in a heterogeneous cluster where the device usage status is masked. o represents the operator sequence number, r represents the device sequence number, mask represents the device usage mask, D represents the set of training devices in the cluster, u represents the currently deployed device selected from the set D, and k represents the partition position in the operator sequence. This represents the minimum and maximum computational cost when the first k operators are deployed to the first r-1 devices after removing device u. This represents the computational cost of deploying the (k+1)th to the 0th operators onto device u.

5. The globally unified large-model pipeline parallel training optimization method according to claim 1, characterized in that, The objective function corresponding to the optimization objective of the phased deployment sub-model. The expression is as follows: In the above formula, Indicates the assembly line stage The mapping relationship between physical training equipment, Let s represent the i-th pipeline stage, and s represent the number of pipeline stages. and Let represent the computation time of the i-th and j-th pipeline stages, respectively. and These represent the communication times related to the i-th and j-th pipeline stages, respectively, and b represents the number of micro-batches. This represents the bubble time generated by s production line stages; The constraints corresponding to the optimization objective of the stage deployment sub-model include memory capacity constraints and stage unique deployment constraints. The memory capacity constraints limit the total memory overhead of pipeline stages deployed to the same training device to not exceed the memory capacity of the training device. The stage unique deployment constraints limit each pipeline stage to be deployed to only one training device.

6. The globally unified large-model pipeline parallel training optimization method according to claim 1, characterized in that, The objective function corresponding to the optimization objective of the strategy scheduling sub-model The expression is as follows: In the above formula, Q represents the candidate scheduling strategy, Y represents the set of candidate scheduling strategies, s represents the number of pipeline stages, and j represents the pipeline stage number. This represents the j-th pipeline stage. and This indicates the start and end times of the j-th pipeline stage. This represents the effective computation time of the o-th training task or operator in the j-th pipeline stage. It represents the sum of the effective computation time of all training tasks or operators within the j-th pipeline stage; The constraints corresponding to the optimization objective of the policy scheduling sub-model include task dependency constraints, which limit the execution order of each micro-batch training task to ensure that any task can only be scheduled and executed after its corresponding parent task has been completed.

7. The globally unified large-model pipeline parallel training optimization method according to claim 1, characterized in that, The total delay in the optimization objective of the globally unified joint solution model includes one or more of the following: the maximum computational load of each pipeline stage in the model partitioning optimization model, the cross-device communication delay between pipeline parallel stages in the stage deployment optimization model, and the total bubble time of the scheduling strategy in the strategy scheduling optimization model. The corresponding objective function expression is as follows: In the above formula, Latency represents the total latency of a single training cycle. Operator The division results This represents the o-th operator. Indicates the assembly line stage The results of the equipment deployment Let Q represent the j-th pipeline stage, Q represent the candidate scheduling strategy, s represent the number of pipeline stages, and b represent the number of micro-batches. This represents the stage computation cost of the j-th pipeline stage. This represents the cross-device communication latency between the j-th pipeline stage and the (j+1)-th pipeline stage. This represents the total bubble time corresponding to scheduling strategy Q.

8. A globally unified large-scale model pipelined parallel training optimization system, comprising interconnected microprocessors and memory, characterized in that, The microprocessor is programmed or configured to perform the steps of the globally unified large model pipeline parallel training optimization method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, The computer program is used to be programmed or configured by a microprocessor to perform the steps of the globally unified large model pipeline parallel training optimization method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, The computer program / instructions are programmed or configured to execute the steps of the globally unified large model pipeline parallel training optimization method according to any one of claims 1 to 7 via a processor.

Images