Model training scheduling method, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a model training scheduling method, electronic equipment and a storage medium. The method comprises the steps that training parameters corresponding to a large language model are obtained, proportion information of training operation of all training devices in the pipeline parallel training process corresponding to the large language model is determined based on the training parameters, and the proportion information is used for representing the proportion between data processing time consumption and data transmission time consumption of the corresponding training operation; determining an initial scheduling solution according to a preset scheduling strategy and the proportion information of the training operation of each training device; determining a mixed integer programming model based on the initial scheduling solution, a preset objective function and a preset constraint condition set; performing scale adaptive solution based on the initial scheduling solution and a mixed integer programming model to obtain a target scheduling solution; and scheduling the pipeline parallel training process based on the target scheduling solution. Based on the scheme of the invention, the overall efficiency of pipeline parallel training can be improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a scheduling method, electronic device, and storage medium for model training. Background Technology

[0002] Large language models have become a representative technology in the field of artificial intelligence. To address the training needs of large language models and overcome the performance bottleneck of single training devices, pipelined parallel training (PP) has been proposed. This method divides the large language model across multiple training devices and collaboratively completes the training task in a pipelined manner. However, during pipelined parallel training, due to the complexity of data flow, idling time is prone to occur on each training device. Therefore, how to improve the overall efficiency of pipelined parallel training has become an urgent technical problem to be solved. Summary of the Invention

[0003] This application provides a model training scheduling method, electronic device, and storage medium to solve the technical problem of how to improve the overall efficiency of pipelined parallel training.

[0004] In a first aspect, embodiments of this application provide a method for scheduling model training, comprising: Obtain the training parameters corresponding to the large language model, and determine the proportion information of training operations of each training device in the pipeline parallel training process corresponding to the large language model based on the training parameters. The proportion information is used to characterize the ratio between the data processing time and the data transmission time of the corresponding training operation. The initial scheduling solution is determined based on the preset scheduling strategy and the proportion of training operations of each training device. Based on the initial scheduling solution, the preset objective function, and the preset set of constraints, a mixed integer programming model is determined. The target scheduling solution is obtained by adaptively solving the problem based on the initial scheduling solution and the mixed integer programming model. Based on the target scheduling solution, the pipeline parallel training process is scheduled.

[0005] In conjunction with the first aspect, among some possible implementation methods, the training parameters corresponding to the large language model are obtained, including: Obtain the model placement order parameters, modeling granularity parameters, pipeline stage parameters, and micro-batch parameters corresponding to the large language model; The training parameters for the large language model are determined based on the model placement order parameters, modeling granularity parameters, pipeline stage parameters, and micro-batch parameters corresponding to the large language model.

[0006] Combining the first aspect and the above implementation methods, in some possible implementation methods, the initial scheduling solution is determined based on the preset scheduling strategy and the proportion information of training operations of each training device, including: Based on the preset scheduling strategy and the proportion of training operations of each training device, multiple execution stages are determined; The initial scheduling solution is determined based on multiple execution phases.

[0007] Combining the first aspect and the above implementation methods, in some possible implementation methods, based on the initial scheduling solution, the preset objective function, and the preset set of constraints, a mixed-integer programming model is determined, including: Based on the initial scheduling solution, obtain the initialized decision variables; A mixed integer programming model is constructed based on decision variables, a preset objective function, and a preset set of constraints. The preset objective function is used to minimize the total execution time of determining the target scheduling solution.

[0008] Combining the first aspect and the above implementation methods, in some possible implementation methods, the set of preset constraints includes at least one of the following: The first constraint is used to constrain the training operations of forward propagation tasks and backward propagation tasks in the same microbatch to be executed in a predetermined order by using the part of the decision variable that represents the start time of the training operation. The second constraint is used to constrain, through the part of the decision variable that represents the start time of the training operation, that the start time of any training operation of the weight update class task in the same micro-batch is not earlier than the end time of the backpropagation class task with the same operation number. The third constraint is used to constrain the execution times of any two training operations on the same training device to not overlap, by using the part of the decision variable that represents the execution order of the training operations. The fourth constraint is used to ensure that when performing any training operation on any training device, the sum of the video memory consumed by the training operation already performed on the training device and the video memory required for the current training operation does not exceed the preset video memory limit. The fifth constraint is used to ensure that the total execution time minimized by the preset objective function is not less than the completion time of any training operation.

[0009] Combining the first aspect and the above implementation methods, in some possible implementation methods, the first constraint is also used to constrain the following time relationship through the part of the decision variable that represents the start time of the training operation: For the same forward propagation task or the same backward propagation task in the same micro-batch, the start time of the next training operation shall not be earlier than the sum of the execution time and communication time of the previous training operation. For the same micro-batch, the start time of the first training operation of a backpropagation task is no earlier than the sum of the execution time and communication time of the last training operation of a forward propagation task.

[0010] Combining the first aspect and the above implementation methods, in some possible implementation methods, the target scheduling solution is obtained by adaptively solving the initial scheduling solution and the mixed-integer programming model, including: If the number of constraints in the mixed integer programming model does not exceed the preset scale threshold, the mixed integer programming model is assigned a first assigned model based on the initial scheduling solution, the first assigned model is solved to obtain the first scheduling solution, and the first scheduling solution is determined as the target scheduling solution. If the number of constraints in the mixed integer programming model exceeds the preset size threshold, the size of the assigned mixed integer programming model is reduced to obtain a reduced mixed integer programming model. Based on the initial scheduling solution, the reduced mixed integer programming model is assigned a second assigned model. The second assigned model is solved and its size is expanded to obtain a second scheduling solution, and the second scheduling solution is determined as the target scheduling solution.

[0011] Combining the first aspect and the above implementation methods, in some possible implementation methods, the first assignment model is obtained by assigning values ​​to the mixed-integer programming model based on the initial scheduling solution, including: Based on the proportion information of training operations of each training device and the execution order of training operations of each training device in the initial scheduling solution, determine the initial value of the start time variable of training operations of each training device. Based on the initial values ​​of the start time variable of the training operation of each training device, the decision variables in the mixed integer programming model are assigned values ​​to obtain the first assignment model; The reduced mixed-integer programming model is assigned values ​​based on the initial scheduling solution to obtain the second assigned model, including: Based on the proportion of training operations of each training device and the execution order of the corresponding part of the initial scheduling solution to the reduced mixed integer programming model, the initial value of the start time variable of the training operation of each training device is determined. Based on the initial values ​​of the start time variable of the training operation of each training device, the decision variables in the reduced mixed integer programming model are assigned values ​​to obtain the second assigned model.

[0012] Combining the first aspect and the above implementation methods, in some possible implementation methods, the scaled-down mixed-integer programming model after assignment is used to obtain a scaled-down mixed-integer programming model, including: Perform at least one of the first scale reduction operation, the second scale reduction operation, and the third scale reduction operation on the assigned mixed integer programming model in sequence until the number of constraints of the reduced mixed integer programming model is less than or equal to the preset scale threshold, and obtain the reduced mixed integer programming model. The first scale reduction operation is: fixing the training operations belonging to the first type of execution phase in the mixed integer programming model after assignment; the second scale reduction operation is: selecting some tasks from the initial scheduling solution to reconstruct the mixed integer programming model after assignment, so that there is at least one training operation belonging to the first type of execution phase on each training device; the third scale reduction operation is: solving only the training operations belonging to the second type of execution phase with a preset proportion. The first type of execution phase and the second type of execution phase are different execution phases.

[0013] Combining the first aspect and the above implementation methods, in some possible implementation methods, the second assignment model is solved and its scale is expanded to obtain the second scheduling solution, including: Solve the second assignment model to obtain the reduced scheduling solution; Extract the first scheduling information of each training operation belonging to the first type of execution phase from the reduced scheduling solution; Extract the second scheduling information of each training operation belonging to the first type of execution phase from the initial scheduling solution; From the reduced scheduling solution, extract the third scheduling information of each training operation after the first type of execution stage, and update the task index of each training operation in the third scheduling information based on the preset index offset to obtain the updated third scheduling information. Based on the first scheduling information, the second scheduling information, and the updated third scheduling information, a second scheduling solution is generated.

[0014] Combining the first aspect and the above implementation methods, in some possible implementation methods, the preset scheduling strategy is the DualPipeV scheduling strategy. The first type of execution phase includes an overlapping phase, and the second type of execution phase includes a reverse-overlapping phase, an interleaved reverse phase, and a cooling phase. The overlapping phase is used to characterize the phase in which training operations for forward propagation tasks and training operations for backward propagation tasks are executed alternately on the training device to mask the time-consuming data transmission phase. The "one-back-propagation-one-overlapping phase" is used to characterize the phase in which the training operations of a single backpropagation class task and the training operations of a single forward propagation class task are executed concurrently on the training device. The staggered backpropagation phase is used to characterize the phase in which training operations for backpropagation-like tasks are performed in an alternating manner on the training device. The cooling phase is used to characterize the training operations of backpropagation-like tasks executed sequentially on the training device to complete the pipelined parallel training process.

[0015] Secondly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the scheduling method for model training of the first aspect.

[0016] Thirdly, embodiments of this application provide a non-transitory computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the scheduling method for model training in the first aspect.

[0017] The model training scheduling method, electronic device, and storage medium provided in this application first obtain the training parameters corresponding to the large language model, and then determine the proportion information of the training operations of each training device in the pipelined parallel training process corresponding to the large language model based on the training parameters. The proportion information is used to characterize the ratio between the data processing time and the data transmission time of the corresponding training operation, thereby introducing real time ratio data. Subsequently, according to the preset scheduling strategy and the proportion information of the training operations of each training device, an initial scheduling solution is determined, and the generation quality of the initial scheduling solution is optimized using the proportion information. Next, based on the initial scheduling solution, the preset objective function, and the preset constraint set, a mixed integer programming model is determined, and the initial scheduling solution is used as the basis for constructing the mixed integer programming model to ensure the accuracy of the model. Afterwards, based on the initial scheduling solution and the mixed integer programming model, scale adaptive solution is performed to obtain the target scheduling solution. The scale adaptive mechanism automatically adjusts the solution strategy when the problem scale is abnormal. Finally, based on the target scheduling solution, the pipelined parallel training process is scheduled to achieve optimized execution. Thus, by introducing the ratio between the actual data processing time and the data transmission time to optimize the generation of scheduling solutions, and by using a mixed integer programming model and a scale-adaptive solution mechanism to ensure the accuracy of the solution, a target scheduling solution that adapts to the ratio information and solves problems of various scales is obtained, thereby improving the overall efficiency of pipeline parallel training. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is an example schematic diagram of a naive pipeline parallel scheduling strategy provided in the embodiments of this application; Figure 2This is a schematic diagram illustrating an example of the virtual pipeline parallel scheduling strategy provided in this application embodiment; Figure 3 This is an example schematic diagram of the ZeroBubble-H1 scheduling strategy provided in the embodiments of this application; Figure 4 This is an example schematic diagram of the ZeroBubble-H2 scheduling strategy provided in the embodiments of this application; Figure 5 This is a schematic diagram illustrating an example of the DualPipe scheduling strategy provided in an embodiment of this application; Figure 6 This is a schematic diagram illustrating an example of all-to-all communication provided in an embodiment of this application; Figure 7 This is a schematic diagram illustrating an example of the DualPipeV scheduling strategy provided in an embodiment of this application. Figure 8 This is a flowchart illustrating the scheduling method for model training provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] Large language models have become a representative technology in the field of artificial intelligence. To address the training needs of large language models and overcome the performance bottleneck of single training devices, pipelined parallel training has been proposed. This approach involves splitting the large language model across multiple training devices and collaboratively completing the training task in a pipeline manner. Several scheduling strategies related to pipelined parallel training are provided below.

[0022] For example, one relevant strategy is the Naive Pipeline Parallel (NPP) scheduling strategy. Please refer to... Figure 1 This strategy involves forward and backward propagation of the entire batch of data sequentially according to the training device stages, with the optimizer then updating the parameters synchronously. Figure 1The execution order of training operations on the timeline is shown: from left to right, F0 (Forward) represents the forward propagation training operation; B0 (Backward) represents the backward propagation training operation; and Update represents the parameter update training operation. The entire process follows the order of forward propagation, backward propagation, and parameter update, processing data batch by batch without micro-batch partitioning. It can be seen that the forward propagation training operation is executed sequentially, the backward propagation training operation is executed sequentially, and the parameter update training operation is executed sequentially. This results in the training operations and communication operations being executed serially, leading to significant idling time between different training devices and low training efficiency.

[0023] For example, one relevant strategy is the Virtual Pipeline Parallel (VPP) scheduling strategy. Please refer to... Figure 2 This strategy, without increasing the number of training devices, further subdivides the model into more virtual stages, sacrificing communication volume for a lower cavitation ratio. Its execution flow includes three stages: Warmup, Steady, and Cooldown. The numbers in the diagram represent the microbatch numbers or sequences, and the color blocks distinguish the training operation types of the training devices. The red box highlights the stable 1F1B pipeline core execution stage, which consists of one forward propagation and one backward propagation. During the warm-up execution phase, each training device sequentially performs forward propagation training operations. For example, training device 1 first performs F1-F4, and training device 2 then performs F1-F4, etc. During the stable execution phase, each training device alternately performs forward propagation training operations and backward propagation training operations. For example, training device 1 performs F8, and training device 2 performs B7, etc., forming an overlapping execution mode of one forward propagation plus one backward propagation. During the cooling execution phase, each training device sequentially performs backward propagation training operations. For example, training device 1 performs B1-B8, and training device 2 performs B1-B8, etc., to reduce the idling time of the training devices.

[0024] For example, one relevant strategy is the ZeroBubble-H1 scheduling strategy. Please refer to... Figure 3This strategy splits the entire batch into multiple micro-batches (as shown in numbers 1-8 in the figure) and divides backpropagation into two parts: B (backpropagation training operation) and W (weight update training operation), while F is the forward propagation training operation. In the pipeline, four training devices, Devices 1 to 4, alternately execute the forward propagation training operation, the backpropagation training operation, and the weight update training operation. For example, Device 1 executes F1-F4 first, followed by B1-B8 and W1-W5, etc., and finally performs the Optimizer step (optimizer step training operation) uniformly, thereby reducing the idling time of the training devices.

[0025] For example, one relevant strategy is the ZeroBubble-H2 scheduling strategy. Please refer to... Figure 4 This strategy, based on the ZeroBubble-H1 scheduling strategy, implements asynchronous optimizer updates. Specifically, the attached diagram shows the training operation sequence of four training devices, Device1 to Device4, on the timeline. Each training device executes forward propagation training operations, backpropagation training operations, and weight update training operations sequentially. The F, B, and W training operations correspond to microbatch numbers 1-8, respectively. Each training device executes these training operations alternately in the pipeline; for example, Device1 executes F1-F7 first, then B1-F8, then W1-F8, and then independently executes the optimizer step training operation. Simultaneously, the optimizer step training operation is executed asynchronously across different training devices. That is, after completing the weight update training operation for the corresponding microbatch, each training device independently updates the optimizer, avoiding synchronous waiting and further optimizing the training operation arrangement on the training devices, thus reducing idling time.

[0026] For example, one relevant strategy is the DualPipe scheduling strategy. Please refer to... Figure 5This strategy creates a stable execution phase in the pipeline called overlap by storing two copies of the model on the training devices, one in ascending order of training device number and the other in reverse order. This overlap phase involves the overlapping execution of forward and backward propagation training operations. In the diagram, the orange and green blocks represent overlapped forward and backward operations, masking all-to-all communication during hybrid expert model training. On the timeline of the eight training devices (Device0-Device7), orange blocks represent forward propagation training operations, green blocks represent backward propagation training operations, light green blocks represent input backward propagation training operations, and blue blocks represent weight backward propagation training operations. During the overlap phase, forward and backward propagation training operations are interleaved to mask all-to-all communication, significantly reducing bubbling time. Further, please combine... Figure 6 The Computation layer performs computational training operations, including MLP backward training, MLP weight update training, MLP forward training, attention backward training, attention weight update training, and attention forward training. The Communication layer handles communication training operations, including distributing forward training, distributing backward training, aggregating forward training, PP(F) (all2all forward training), PP(B) (all2all backward training), and aggregating backward training. During the overlapping execution phase, the key design is the overlapping execution of a forward propagation plus a set of backward propagation weights. For example, MLP forward training and all2all forward training are performed in parallel, as are MLP backward training and all2all backward training. This ensures that all-to-all communication training operations, such as PP operations, are covered by computation time, avoiding communication waiting time and significantly reducing idle time.

[0027] For example, one relevant strategy is the DualPipeV scheduling strategy. Please refer to... Figure 7This strategy, an improvement on the DualPipe scheduling strategy, only requires storing one copy of the model and can still generate overlap (i.e., overlapping execution phases) to interleave forward and backward propagation training operations to mask all-to-all communication. This strategy executes in seven phases sequentially across four training devices (Device0 to Device3): warmup (preheating execution phase), interleaved_forward (interleaved forward propagation execution phase), 1b1w1f (one forward-weight-one backward propagation execution phase), overlapping execution phase, 1b1overlap (one backward-overlapping execution phase), interleaved_backward (interleaved backward propagation execution phase), and cooldown (cooling-down execution phase). In the attached diagram, different colored blocks on each training device represent training operations: orange blocks represent forward propagation training operations, green blocks represent backward propagation training operations, light green blocks represent input backward propagation training operations, blue blocks represent weight backward propagation training operations, and alternating orange and green blocks represent overlapping forward and backward propagation training operations. The numbers 0-9 on each block represent the microbatch number. For example, Device0 performs forward propagation training operations 0-5 during the warm-up execution phase, mixed training operations 0-7 during the one-forward-one-weight-one-backward execution phase, overlapping training operations 4-9 during the overlapping execution phase, mixed training operations 6-9 during the one-backward-one-overlap execution phase, backpropagation training operations 6-9 during the staggered backpropagation execution phase, and cooling training operations 6-9 during the cooling execution phase. As the number of micro-batches increases, only the overlapping execution phase expands, while the number of training operations in other execution phases remains unchanged. However, during pipelined parallel training, due to the complexity of data processing and data transmission, communication waits and computational idles can easily occur between training devices.

[0028] Most of the aforementioned strategies are designed under an idealized model where the number of forward propagation training operations, weight update training operations, and backpropagation training operations are all 1. This differs significantly from the actual computation time, communication time, and memory usage of each training operation during training. Especially in hybrid expert model training scenarios, when all-to-all communication becomes a key factor affecting training speed, if training tasks such as forward propagation training operations, weight update training operations, and backpropagation training operations are further broken down into more granular training operations such as attention layer computation training operations, all-to-all communication distribution training operations, multilayer perceptron layer computation training operations, and all-to-all communication aggregation training operations, existing strategies struggle to achieve efficient scheduling that balances data processing and data transmission at this granular level. Even virtual pipeline parallel scheduling strategies and DualPipe scheduling strategies can only mask all-to-all communication during the stable execution phase of their pipelines, and cannot globally optimize the overlap of data processing and data transmission throughout the entire training process. They also cannot directly consider the memory constraints of the training device during the scheduling strategy formulation process, and cannot further optimize and adjust memory usage based on the determined scheduling scheme.

[0029] In summary, improving the overall efficiency of pipelined parallel training has become an urgent technical problem to be solved.

[0030] To address the aforementioned issues, the solution provided in this application mainly includes: First, obtaining the training parameters corresponding to the large language model, and determining the proportion information of training operations of each training device during the pipelined parallel training process corresponding to the large language model based on the training parameters. The proportion information is used to characterize the ratio between the data processing time and data transmission time of the corresponding training operation, thereby introducing real time ratio data. Subsequently, determining the initial scheduling solution based on the preset scheduling strategy and the proportion information of training operations of each training device, and optimizing the generation quality of the initial scheduling solution using the proportion information. Next, determining the mixed integer programming model based on the initial scheduling solution, the preset objective function, and the preset constraint set, using the initial scheduling solution as the basis for constructing the mixed integer programming model to ensure model accuracy. Afterward, performing scale-adaptive solving based on the initial scheduling solution and the mixed integer programming model to obtain the target scheduling solution, and automatically adjusting the solution strategy when the problem scale is abnormal through the scale-adaptive mechanism. Finally, scheduling the pipelined parallel training process based on the target scheduling solution to achieve optimized execution. Thus, by introducing the ratio between the actual data processing time and the data transmission time to optimize the generation of scheduling solutions, and by using a mixed integer programming model and a scale-adaptive solution mechanism to ensure the accuracy of the solution, a target scheduling solution that adapts to the ratio information and solves problems of various scales is obtained, thereby improving the overall efficiency of pipeline parallel training.

[0031] The following will provide a detailed description of the model training scheduling method provided in the embodiments of this application.

[0032] Please see Figure 8 , Figure 8 This is a flowchart illustrating a model training scheduling method provided in an embodiment of this application. Figure 8 As shown, the method in this application embodiment may include the following steps S101-S105.

[0033] S101, obtain the training parameters corresponding to the large language model, and determine the proportion information of training operations of each training device in the pipeline parallel training process corresponding to the large language model based on the training parameters. The proportion information is used to characterize the ratio between the data processing time and the data transmission time of the corresponding training operation.

[0034] Specifically, the first step is to obtain the training parameters corresponding to the large language model. The large language model refers to a deep learning neural network model trained on massive amounts of text data for understanding and generating human language. The training parameters corresponding to the large language model refer to the parameters used to configure the model and execution environment during training. For example, training parameters may include at least one of the following: model placement order parameters, modeling granularity parameters, pipeline stage parameters, and micro-batch parameters.

[0035] Regarding this step, in some possible implementations, configuration information describing the model structure and hardware characteristics can be obtained through relevant configuration interfaces, and the received configuration information can be used as training parameters. In other possible implementations, a predefined training parameter file can be read through relevant parsing functions, and the parsed configuration data can be used as training parameters.

[0036] Furthermore, it is necessary to determine the proportion of training operations of each training device in the pipelined parallel training process corresponding to the large language model based on the training parameters. Here, each training device in the pipelined parallel training process corresponding to the large language model refers to the device participating in parallel computation; the training operations of each training device refer to the atomic operations contained in different types of tasks such as forward propagation tasks, back propagation tasks, and weight update tasks; the proportion of training operations of each training device is used to characterize the ratio between the data processing time and the data transmission time of the corresponding training operation; the data processing time of the training operation refers to the execution time of the operation on the training device; the data transmission time of the training operation refers to the communication time of the operation on the interconnection network between the training devices; the ratio between the data processing time and the data transmission time of the training operation refers to the numerical ratio of the data processing time to the data transmission time.

[0037] Regarding this step, in some possible implementations, performance sampling processing can be performed on the training operations to collect relevant time data, and mathematical operations can be performed on the relevant time data to obtain the proportion information of training operations for each training device. In some possible implementations, estimation processing can be performed based on the hardware characteristics in the training parameters, and the analysis results can be converted into the proportion information of training operations for each training device.

[0038] S102, determine the initial scheduling solution based on the preset scheduling strategy and the proportion information of training operations of each training device.

[0039] Specifically, in order to establish a basic feasible solution for subsequent optimization, an initial scheduling solution needs to be determined based on the preset scheduling strategy and the proportion of training operations on each training device. The preset scheduling strategy refers to predefined rules or methods for generating the order of operation execution; for example, the preset scheduling strategy could be the DualPipeV scheduling strategy or other heuristic scheduling algorithms. The initial scheduling solution refers to the execution order and time allocation scheme of training operations on each training device, which can be represented as a set of operation sequences and time parameters.

[0040] Regarding this step, some possible implementations involve using a pre-defined scheduling strategy to divide the training operations into stages, and using the sequence of operations corresponding to the generated execution stages as the initial scheduling solution. Other possible implementations involve heuristic scheduling based on the proportion of training operations, using the generated operation sequence as the initial scheduling solution.

[0041] S103. Based on the initial scheduling solution, the preset objective function, and the preset set of constraints, determine the mixed integer programming model.

[0042] Specifically, to transform the scheduling problem into a mathematical optimization form, a mixed-integer programming model needs to be determined based on the initial scheduling solution, a pre-defined objective function, and a pre-defined set of constraints. The pre-defined objective function refers to a mathematical index used to measure the quality of the scheduling scheme; for example, it could be a linear function that minimizes the maximum completion time. The pre-defined set of constraints refers to the set of restrictions that must be satisfied during the scheduling process; for example, it could include at least one type of constraint such as constraints on operation timing dependencies, constraints on device resource conflicts, and constraints on memory usage. The mixed-integer programming model is a mathematical model used to solve the optimization problem, where the decision variables include both continuous and integer variables.

[0043] Regarding this step, in some possible implementations, variable definition can be performed based on the initial scheduling solution, and a mixed-integer programming model can be obtained by combining a preset objective function and a preset set of constraints. In other possible implementations, constraint transformation can be performed on the initial scheduling solution, and the transformed mathematical constraints can be combined with the preset objective function to obtain the mixed-integer programming model.

[0044] S104, based on the initial scheduling solution and the mixed integer programming model, performs adaptive scaling to obtain the target scheduling solution.

[0045] Specifically, to balance solution accuracy and computational efficiency, a scale-adaptive solution is needed based on the initial scheduling solution and the mixed-integer programming model to obtain the target scheduling solution. Here, scale-adaptive solution refers to the process of dynamically adjusting the solution strategy according to the problem size; the target scheduling solution refers to the optimized scheduling scheme that satisfies the constraints, which can be expressed as an exact solution in terms of operation timing.

[0046] Regarding this step, some possible implementations include performing a size assessment on the mixed-integer programming model, solving it directly based on the assessment result, or simplifying the model and using the solution as the target scheduling solution. Other possible implementations include performing a size reduction process based on the initial scheduling solution, solving the reduced problem, and then performing a size expansion process, using the expanded result as the target scheduling solution.

[0047] S105 schedules the pipeline parallel training process based on the target scheduling solution.

[0048] Specifically, in order to achieve automated execution control of the training process, it is necessary to schedule the pipelined parallel training process based on the target scheduling solution. Here, scheduling refers to the process of allocating training operations to training devices for execution according to the target scheduling solution.

[0049] Regarding this step, in some possible implementations, the target scheduling solution can be processed by instruction conversion, and the generated executable scheduling control instructions can be distributed to the training device to implement scheduling.

[0050] In this embodiment, firstly, the training parameters corresponding to the large language model are obtained, and based on the training parameters, the proportion information of the training operations of each training device in the pipelined parallel training process corresponding to the large language model is determined. The proportion information is used to characterize the ratio between the data processing time and the data transmission time of the corresponding training operation, thereby introducing real time ratio data. Subsequently, according to the preset scheduling strategy and the proportion information of the training operations of each training device, an initial scheduling solution is determined, and the generation quality of the initial scheduling solution is optimized using the proportion information. Next, based on the initial scheduling solution, the preset objective function, and the preset constraint set, a mixed integer programming model is determined, and the initial scheduling solution is used as the basis for constructing the mixed integer programming model to ensure the accuracy of the model. Afterwards, based on the initial scheduling solution and the mixed integer programming model, scale-adaptive solving is performed to obtain the target scheduling solution. The scale-adaptive mechanism automatically adjusts the solution strategy when the problem scale is abnormal. Finally, based on the target scheduling solution, the pipelined parallel training process is scheduled to achieve optimized execution. Thus, by introducing the ratio between the actual data processing time and the data transmission time to optimize the generation of scheduling solutions, and by using a mixed integer programming model and a scale-adaptive solution mechanism to ensure the accuracy of the solution, a target scheduling solution that adapts to the ratio information and solves problems of various scales is obtained, thereby improving the overall efficiency of pipeline parallel training.

[0051] In one embodiment, the step of "obtaining the training parameters corresponding to the large language model" can be further refined and may include the following steps: Obtain the model placement order parameters, modeling granularity parameters, pipeline stage parameters, and micro-batch parameters corresponding to the large language model; The training parameters for the large language model are determined based on the model placement order parameters, modeling granularity parameters, pipeline stage parameters, and micro-batch parameters.

[0052] Specifically, the first step is to obtain the model placement order parameters, modeling granularity parameters, pipeline stage parameters, and micro-batch parameters corresponding to the large language model. The model placement order parameters define the allocation order of the various computational stages of the model within the training pipeline; the modeling granularity parameters define the smallest unit level of the training operation decomposition; the pipeline stage parameters define the total number of stages or the staged strategy in parallel pipeline training; and the micro-batch parameters define the number or size of micro-batches into which a data batch is split.

[0053] Regarding this step, in some possible implementations, the model placement order parameters, modeling granularity parameters, pipeline stage parameters, and micro-batch parameters corresponding to the large language model can be obtained by reading a predefined configuration file or interface, and the read configuration data can be directly used as the model placement order parameters, modeling granularity parameters, pipeline stage parameters, and micro-batch parameters corresponding to the large language model.

[0054] Based on this, the training parameters corresponding to the large language model are determined according to the model placement order parameters, modeling granularity parameters, pipeline stage parameters, and micro-batch parameters.

[0055] Regarding this step, in some possible implementations, parameter validation and aggregation can be performed based on the model placement order parameters, modeling granularity parameters, pipeline stage parameters, and micro-batch parameters corresponding to the acquired large language model, and the processed parameter set can be used as the training parameters corresponding to the large language model.

[0056] In this embodiment, by acquiring the model placement order parameters, modeling granularity parameters, pipeline stage parameters, and micro-batch parameters corresponding to the large language model, and determining the training parameters accordingly, this process provides a structured data foundation for subsequently determining the initial scheduling solution. Specifically, the model placement order parameters provide a basis for determining the allocation relationship of training operations on each training device; the modeling granularity parameters define the basic building blocks of the subsequent training operations; and the pipeline stage parameters and micro-batch parameters jointly define the scope and basic structure of the scheduling problem. In summary, this embodiment ensures that the generation of the initial scheduling solution has a clear input basis, laying the foundation for subsequent steps and thus supporting the goal of scheduling the pipelined parallel training process under real-world data proportions.

[0057] In one embodiment, the step of "determining the initial scheduling solution based on the preset scheduling strategy and the proportion information of training operations of each training device" can be further refined and may include the following steps: Based on the preset scheduling strategy and the proportion of training operations of each training device, multiple execution stages are determined; The initial scheduling solution is determined based on multiple execution phases.

[0058] Specifically, considering the complexity of operation scheduling and the need for communication and computational collaboration during pipelined parallel training, this embodiment proposes a scheduling solution generation scheme based on stage partitioning. First, to structurally define the temporal relationships and resource allocation logic of training operations, multiple execution stages need to be determined based on the preset scheduling strategy and the proportion of training operations for each training device. An execution stage refers to a functional execution unit that divides the training operations of the training device according to the time dimension. Each execution stage contains a group of training operations with similar temporal characteristics or resource constraints. Taking the virtual pipelined parallel scheduling strategy as an example, the multiple execution stages are represented as a warm-up execution stage, a stable execution stage, and a cool-down execution stage; taking the DualPipeV scheduling strategy as an example, the multiple execution stages are represented as a warm-up execution stage, an interleaved forward propagation execution stage, a forward-weighted-backward execution stage, an overlapping stage, a backward-overlapping execution stage, an interleaved backpropagation execution stage, and a cool-down execution stage. It is understood that, depending on the preset scheduling strategy, the multiple execution stages will also differ in content and quantity, which will not be listed here.

[0059] Regarding this step, in some possible implementations, the proportion information of training operations of each training device can be used as a basis. This proportion information represents the ratio between the data processing time and the data transmission time of the training operation. Combined with the stage division rules of the preset scheduling strategy, the training operation type and sorting priority corresponding to each execution stage can be calculated. Based on the training operation type and sorting priority, multiple execution stages can be determined.

[0060] Based on this, the initial scheduling solution is further determined based on multiple execution phases.

[0061] Regarding this step, in some possible implementations, for each execution stage, the time allocation parameters of the training operations within the stage can be calculated based on the proportion information of the training operations, and multiple execution stages can be merged to generate an initial scheduling solution based on the time allocation parameters.

[0062] In this embodiment, a stage-based mechanism decouples the complex scheduling problem into functional, stage-based sub-problems. By combining proportional information, the training operations are ordered within each execution stage. Adjusting the order of operations allows for the parallel execution of training operations with high data processing time and those involving data transmission time. The resulting initial scheduling solution satisfies global timing constraints. Furthermore, because data processing and data transmission are coordinated during operation ordering, a better starting point is provided for solving the subsequent mixed-integer programming model, thereby reducing the convergence difficulty of the overall optimization process.

[0063] In one embodiment, the step of "determining the mixed-integer programming model based on the initial scheduling solution, the preset objective function, and the preset set of constraints" can be further refined and may include the following steps: Based on the initial scheduling solution, obtain the initialized decision variables; A mixed-integer programming model is constructed based on decision variables, a preset objective function, and a preset set of constraints. The preset objective function is used to minimize the total execution time of determining the target scheduling solution.

[0064] Specifically, considering the complexity of operation scheduling and the diversity of resource constraints during pipeline parallel training, this embodiment proposes a scheme to transform the scheduling problem into an optimizable mixed integer programming problem through mathematical modeling.

[0065] First, based on the initial scheduling solution, we need to obtain the initial decision variables. Initialization refers to the process of assigning values ​​to the decision variables based on the initial scheduling solution; the initial decision variables are the set of variables representing the start time, execution order, and completion time of the training operations.

[0066] Regarding this step, in some possible implementations, the initial value of the start time variable of each training operation can be calculated by parsing the operation execution order of each training device in the initial scheduling solution and combining it with the proportion information of the training operations. The calculated initial value of the start time variable can then be used as the initial decision variable.

[0067] Building upon this foundation, a mixed-integer programming model is further constructed based on the decision variables, the preset objective function, and the preset set of constraints. The construction refers to the process of combining the decision variables, the preset objective function, and the preset set of constraints into a mathematical optimization model.

[0068] Regarding this step, in some possible implementations, the decision variables, the preset objective function, and the preset set of constraints can be input into the mixed integer programming solver, and the mixed integer programming model can be generated by the mixed integer programming solver.

[0069] In this embodiment, the initial scheduling solution provides a basis for assigning values ​​to decision variables and an initial point for subsequent optimization. A preset objective function, used to minimize the total execution time, is combined to set the optimization objective. Simultaneously, a set of preset constraints is integrated to ensure the scheduling scheme meets limitations regarding operation timing, device resource conflicts, and memory usage. The mixed-integer programming model, constructed from the decision variables, the preset objective function, and the set of preset constraints, transforms the pipeline parallel training scheduling problem into a standard mathematical optimization problem. This provides an accurate and computable mathematical foundation for adaptive scaling solutions, enabling the search for a target scheduling solution that may be superior to the initial scheduling solution.

[0070] In one embodiment, the set of preset constraints includes at least one of the following: The first constraint is used to constrain the training operations of forward propagation tasks and backward propagation tasks in the same microbatch to be executed in a predetermined order by using the part of the decision variable that represents the start time of the training operation. The second constraint is used to constrain, through the part of the decision variable that represents the start time of the training operation, that the start time of any training operation of the weight update class task in the same micro-batch is not earlier than the end time of the backpropagation class task with the same operation number. The third constraint is used to constrain the execution times of any two training operations on the same training device to not overlap, by using the part of the decision variable that represents the execution order of the training operations. The fourth constraint is used to ensure that when performing any training operation on any training device, the sum of the video memory consumed by the training operation already performed on the training device and the video memory required for the current training operation does not exceed the preset video memory limit. The fifth constraint is used to ensure that the total execution time minimized by the preset objective function is not less than the completion time of any training operation.

[0071] Specifically, considering the inherent operational timing dependencies, device resource competition, and memory limitations in the pipeline parallel training process, this embodiment proposes a scheme to formalize these inherent constraints into mathematical constraints and incorporate them into a mixed integer programming model to guide the solution process.

[0072] It should be noted that one or more of the first, second, third, fourth, and fifth constraints can be selected to construct the preset constraint set. The following is a detailed explanation of the constraint variables that may be included in the preset constraint set: The variables and indices are defined as follows: : Indicates a micro-batch index; : Represents the task type index, where Represents forward propagation tasks (using) express), Represents backpropagation tasks (using) express), Represents weight update tasks (using) express); : Represents the operation index within a task; : Indicates the training device index.

[0073] The first constraint is used to ensure that the training operations within the forward and backward propagation tasks of the same micro-batch are executed sequentially according to a preset training operation index order, and that the start time of the first training operation of the backward propagation task is no earlier than the end time of the last training operation of the forward propagation task. Its purpose is to ensure that the data dependencies between training operations within the micro-batch are satisfied, thus guaranteeing the correctness of the training process.

[0074] For example, the formula for the first constraint condition is expressed as: ; ; The variables and their corresponding representations are as follows: :express The The first class of tasks The start time of each training operation belong ; :express The The first class of tasks The execution time of each training operation. belong ; :express The The first class of tasks Communication time for each training operation belong ; :express The The first class of tasks The start time of each training operation belong ; :express Forward propagation tasks The last training operation () The start time of ) :express Forward propagation tasks The last training operation () ) execution time; :express Forward propagation tasks The last training operation () The communication time; :express Backpropagation tasks ( The first training operation () The start time of ).

[0075] The second constraint: used to constrain the weight update tasks within the same microbatch. Training The start time of the operation shall not be earlier than the [number]th [number]th [task] of the backpropagation type within the same micro-batch. The end time of each training operation. Its purpose is to ensure that the corresponding gradient calculation has been completed before the weight update is performed, thus satisfying the update logic of the training algorithm.

[0076] For example, the formula for the second constraint is expressed as: ; The variables and their corresponding representations are as follows: :express Backpropagation tasks ( ) The start time of each training operation; :express Backpropagation tasks ( ) The execution time of each training operation; :express Weight update tasks ( ) The start time of each training operation.

[0077] The third constraint: This constraint ensures that the execution time periods of any two different training operations allocated on the same training device do not overlap. Its purpose is to avoid computational resource conflicts on a single training device and ensure the determinism of hardware execution.

[0078] For example, the formula for the third constraint is expressed as: ; ; The variables and their corresponding representations are as follows: : Represents a positive number of a specified size; : Indicates training equipment superior, The The first class of tasks Is each training operation in The The first class of tasks Executed before each training operation - Decision variables; :express The The first class of tasks The start time of each training operation; :express The The first class of tasks The execution time of each training operation; :express The The first class of tasks The start time of each training operation; :express The The first class of tasks The execution time of each training operation.

[0079] The fourth constraint ensures that, when performing any training operation on any training device, the sum of the video memory already occupied on that device and the additional video memory required for the current training operation does not exceed a preset video memory limit. Its purpose is to ensure that the scheduling scheme meets the hardware's video memory resource limitations and avoids training failure due to video memory overflow.

[0080] For example, the formula for the fourth constraint is expressed as: ; The variables and their corresponding representations are as follows: : Indicates training equipment Above, all in training operations The total amount of additional GPU memory required for the previously performed training operations; : Indicates training equipment superior, The The first class of tasks Is each training operation in The The first class of tasks Executed before each training operation - Decision variables; :express The The first class of tasks The additional GPU memory required for each training operation; :express The The first class of tasks The additional GPU memory required for each training operation; : Indicates the preset maximum video memory limit of the training device.

[0081] The fifth constraint: This constraint ensures that the total execution time minimized by the preset objective function is not less than the completion time of any training operation. Its purpose is to ensure that the optimization objective... It can accurately characterize the end time of the entire pipeline parallel training process.

[0082] For example, the formula for the fifth constraint is expressed as: ; The variables and their corresponding representations are as follows: :express The The first class of tasks The start time of each training operation; :express The The first class of tasks The execution time of each training operation; : An auxiliary variable representing the maximum completion time of the entire pipeline's parallel training process.

[0083] In this embodiment, the first and second constraints define the temporal dependencies between different types of tasks within the same micro-batch; the third constraint ensures the exclusivity of operations on the same training device; the fourth constraint introduces limitations on GPU memory resources; and the fifth constraint aligns the optimization objective with the actual completion time. Any combination of these five constraints constitutes a complete mathematical description of the pipelined parallel training scheduling problem, ensuring that the target scheduling solution obtained by the mixed-integer programming model is mathematically optimal under specific conditions and physically feasible and correct.

[0084] In one embodiment, the first constraint is further used to constrain the following temporal relationship by using the portion of the decision variable that represents the start time of the training operation: For the same forward propagation task or the same backward propagation task in the same micro-batch, the start time of the next training operation shall not be earlier than the sum of the execution time and communication time of the previous training operation. For the same micro-batch, the start time of the first training operation of a backpropagation task is no earlier than the sum of the execution time and communication time of the last training operation of a forward propagation task.

[0085] Specifically, considering the operational timing dependencies during pipelined parallel training and all-to-all communication in hybrid expert model scenarios, this embodiment proposes a scheme that formalizes the operation execution order through mathematical constraints and supports collaborative communication.

[0086] On the one hand, for the same forward propagation task or the same backward propagation task within the same micro-batch, the start time of the next training operation is no earlier than the sum of the execution time and communication time of the previous training operation. Specifically, this is achieved through decision variables. Representing the start time of training operations, combined with execution time and communication time It constructs operational continuity constraints and supports the interleaved execution of training operations for forward propagation tasks and backpropagation tasks during overlapping execution phases, and establishes a communication coordination mechanism where all-to-all communication is masked by computational time consumption.

[0087] For example, it can be represented as: ; in, express The The first class of tasks The start time of each training operation; This indicates the execution time of the training operation; This indicates the communication time for the training operation; Indicates the start time of the next training operation within the same micro-batch and the same type of task; belong , Represents forward propagation tasks. This represents a backpropagation task.

[0088] On the other hand, for the same micro-batch, the start time of the first training operation of the backpropagation task is no earlier than the sum of the execution time and communication time of the last training operation of the forward propagation task. Specifically, this is achieved by establishing a global temporal dependency between the forward propagation and backpropagation tasks, specifically the data dependency relationship between the first training operation of the backpropagation task and the completion of the last training operation of the forward propagation task.

[0089] For example, it can be represented as: ; in, express The start time of the last training operation in a forward propagation-type task; and These represent the execution time and communication time of the training operation, respectively. This indicates the start time of the first training operation in the same micro-batch backpropagation task.

[0090] In this embodiment, decision variables are used. With execution time Communication time The quantitative constraints first satisfy the sequential execution requirements of training operations at the operational granularity level, ensuring the data dependency between forward propagation and back propagation tasks within the same micro-batch. Second, it supports the interleaved execution of training operations for forward propagation and back propagation tasks in the overlapping execution phase of the mixed integer programming model, so that the computation time of all-to-all communication operations is covered, solving the problem of all-to-all communication affecting training speed in the training scenario of mixed expert models. Finally, by explicitly distinguishing the constraints between execution time and communication time, the mixed integer programming model can solve the problem based on the ratio between the actual data processing time and data transmission time, overcoming the defect that the design assumption that the number of idealized forward propagation tasks, back propagation tasks, and weight update tasks is all 1 differs greatly from the actual training data.

[0091] In one embodiment, the step of "performing adaptive scaling based on the initial scheduling solution and the mixed-integer programming model to obtain the target scheduling solution" can be further refined and may include the following steps: If the number of constraints in the mixed integer programming model does not exceed the preset scale threshold, the mixed integer programming model is assigned a first assigned model based on the initial scheduling solution, the first assigned model is solved to obtain the first scheduling solution, and the first scheduling solution is determined as the target scheduling solution. If the number of constraints in the mixed integer programming model exceeds the preset size threshold, the size of the assigned mixed integer programming model is reduced to obtain a reduced mixed integer programming model. Based on the initial scheduling solution, the reduced mixed integer programming model is assigned a second assigned model. The second assigned model is solved and its size is expanded to obtain a second scheduling solution, and the second scheduling solution is determined as the target scheduling solution.

[0092] Specifically, considering that the solution complexity of mixed integer programming models increases rapidly with the number of constraints, which may lead to excessively long solution times or failure to converge for large-scale problems, this embodiment proposes a scale-adaptive solution scheme that dynamically adjusts the solution strategy based on the number of constraints.

[0093] First, to effectively control the computational resource consumption and time overhead of the solution process, it is necessary to determine whether the number of constraints in the mixed-integer programming model exceeds a preset size threshold. Based on the results of this determination, there are two possible scenarios and corresponding steps to be performed: In one possible scenario, the number of constraints in the mixed-integer programming model does not exceed a preset size threshold, meaning the model's size is within the computational capacity's tolerance and can be solved directly without scaling. In this case, it is necessary to assign values ​​to the mixed-integer programming model based on the initial scheduling solution to obtain a first assigned model. Solving this first assigned model yields a first scheduling solution, which is then determined as the target scheduling solution. Here, the first assigned model refers to the model after assigning values ​​to the decision variables in the mixed-integer programming model based on the initial scheduling solution; the first scheduling solution refers to the optimized scheduling scheme obtained after solving the first assigned model using mixed-integer programming.

[0094] Regarding this step, in some possible implementations, it can be done as follows: Calculate the initial value of the start time variable of the training operation for each training device based on the proportion information of the training operations of each training device and the execution order of the training operations of each training device in the initial scheduling solution; assign values ​​to the decision variables in the mixed integer programming model based on the initial values ​​of the start time variables of the training operations of each training device to obtain the first assigned model; call the mixed integer programming solver to solve the first assigned model to obtain the first scheduling solution; and output the first scheduling solution as the target scheduling solution.

[0095] In one possible scenario, the number of constraints in a mixed-integer programming model exceeds a preset size threshold, indicating that the model is too large. Directly solving it may lead to resource exhaustion or timeouts, necessitating reduction to decrease complexity. In this case, the assigned mixed-integer programming model needs to be scaled down to obtain a reduced model. Based on the initial scheduling solution, the reduced model is then assigned values ​​to obtain a second assigned model. Solving and scaling the second assigned model yields a second scheduling solution, which is then identified as the target scheduling solution. Here, scaling down the assigned mixed-integer programming model refers to reducing the problem size by fixing some operations or reducing the number of tasks. The second assigned model refers to the model after assigning values ​​to the decision variables in the reduced mixed-integer programming model based on the initial scheduling solution. Solving and scaling the second assigned model involves first solving the second assigned model to obtain the reduced scheduling solution, and then expanding this solution to the original problem size to generate a complete scheduling scheme. The second scheduling solution is the final optimized scheduling scheme obtained after solving and scaling.

[0096] Regarding this step, some possible implementations include: performing a first scale reduction operation on the assigned mixed-integer programming model to fix the training operations belonging to the first type of execution phase; assigning values ​​to the reduced mixed-integer programming model based on the initial scheduling solution to obtain a second assigned model; calling the mixed-integer programming solver to solve the second assigned model to obtain the reduced scheduling solution; performing scale expansion on the reduced scheduling solution, including extracting the second scheduling information of each training operation belonging to the first type of execution phase from the initial scheduling solution, and updating the index offset with the third scheduling information of each training operation after the first type of execution phase in the reduced scheduling solution to generate a second scheduling solution; and outputting the second scheduling solution as the target scheduling solution.

[0097] In this embodiment, by comparing the number of constraints in the mixed-integer programming model with a preset size threshold, when the number of constraints does not exceed the preset size threshold, a first scheduling solution is generated by directly solving the initial scheduling solution, ensuring the accuracy of the target scheduling solution. When the number of constraints exceeds the preset size threshold, a combination of size reduction and size expansion is used to transform the original problem into a solvable reduced mixed-integer programming model, and a second scheduling solution matching the size of the original problem is reconstructed through size expansion, thereby ensuring the feasibility of solving in large-scale scenarios and realizing adaptive solutions to pipeline parallel training scheduling problems of different sizes.

[0098] In one embodiment, the above step of "assigning values ​​to the mixed integer programming model based on the initial scheduling solution to obtain the first assigned model" can be further refined and may include the following steps: Based on the proportion information of training operations of each training device and the execution order of training operations of each training device in the initial scheduling solution, determine the initial value of the start time variable of training operations of each training device. Based on the initial values ​​of the start time variable of the training operation of each training device, the decision variables in the mixed integer programming model are assigned values ​​to obtain the first assignment model; Further refining the above step of "assigning values ​​to the reduced mixed-integer programming model based on the initial scheduling solution to obtain the second assigned model" can include the following steps: Based on the proportion of training operations of each training device and the execution order of the corresponding part of the initial scheduling solution to the reduced mixed integer programming model, the initial value of the start time variable of the training operation of each training device is determined. Based on the initial values ​​of the start time variable of the training operation of each training device, the decision variables in the reduced mixed integer programming model are assigned values ​​to obtain the second assigned model.

[0099] Specifically, considering that the solution efficiency of mixed integer programming models depends on the quality of the initial solution, and that the assignment process needs to provide the solver with a starting point close to a feasible solution, this embodiment proposes a scheme to derive the initial values ​​of decision variables based on the initial scheduling solution.

[0100] On the one hand, to ensure that the first assignment model accurately reflects the initial scheduling scheme, it is necessary to determine the initial value of the start time variable of the training operation for each training device based on the proportion information of the training operations of each training device and the execution order of the training operations of each training device in the initial scheduling solution. Here, the execution order of the training operations of each training device in the initial scheduling solution refers to the sequential arrangement of the training operations on each training device as defined in the initial scheduling scheme; the initial value of the start time variable of the training operation for each training device refers to the reference value of the start time of each operation calculated based on the execution order and the ratio of data processing time to data transmission time.

[0101] Regarding this step, in some possible implementations, the training operation sequence of each training device in the initial scheduling solution can be traversed, and the initial value of the start time variable of the training operation of each training device can be calculated by accumulating the data processing time and data transmission time ratio of adjacent operations.

[0102] Furthermore, based on the initial value of the start time variable of the training operation of each training device, the decision variables in the mixed integer programming model are assigned values ​​to obtain the first assigned model.

[0103] Regarding this step, in some possible implementations, the calculated initial value of the start time variable can be assigned to the decision variable representing the start time of the training operation, and the initial value of the decision variable representing the execution order of the training operation can be determined according to the execution order, forming the first assignment model.

[0104] On the other hand, to maintain the effectiveness of the assignment process in the scaling-down scenario, it is necessary to determine the initial value of the start time variable of the training operation for each training device based on the proportion of training operations for each training device and the execution order of the corresponding part of the initial scheduling solution corresponding to the scaled-down mixed-integer programming model. Here, the execution order of the corresponding part of the initial scheduling solution corresponding to the scaled-down mixed-integer programming model refers to the sequence of remaining training operations in the initial scheduling solution that are not fixed; the initial value of the start time variable of the training operation for each training device refers to the time reference value calculated for that remaining training operation sequence.

[0105] Regarding this step, in some possible implementations, a subset of training operations belonging to the reduced mixed-integer programming model can be extracted from the initial scheduling solution, and the initial value of the start time variable can be recalculated according to the execution order and proportion information of this subset.

[0106] Furthermore, based on the initial value of the start time variable of the training operation of each training device, the decision variables in the reduced mixed integer programming model are assigned values ​​to obtain the second assigned model.

[0107] Regarding this step, in some possible implementations, the initial value of the start time variable of the subset computation can be assigned to the decision variable representing the start time of the training operation in the reduced model, while inheriting the value of the decision variable representing the execution order of the training operation in the fixed operation in the initial scheduling solution, thus generating the second assigned model.

[0108] For example, the preset scheduling strategy is the DualPipeV scheduling strategy. The execution order of the training operations of each training device in the initial scheduling solution refers to the operation sequence generated by the DualPipeV scheduling strategy. During the assignment process, it is necessary to ensure that the assignment result satisfies the preset constraint set. Regarding this step, in some possible implementations, the initial value of the start time variable of the training operation of each training device can be calculated by accumulating the data processing time and data transmission time ratio of each operation according to the operation sequence generated by DualPipeV. Furthermore, the calculated initial value of the start time variable can be assigned to the decision variable. And determine the decision variables that satisfy the first and third constraints based on the operation sequence. The initial values. In scaling-down scenarios, such as after the first scaling-down operation, the execution order of the parts of the initial scheduling solution corresponding to the scaled-down mixed-integer programming model refers to the remaining sequence of operations in the initial scheduling solution that were not fixed. Regarding this step, in some possible implementations, a subset of operations belonging to the scaled-down mixed-integer programming model can be extracted from the initial scheduling solution, and the initial values ​​of the start time variables can be recalculated according to the operation sequence within the subset. Furthermore, the initial values ​​of the start time variables calculated from the subset can be assigned to the decision variables of the scaled-down model. Simultaneously, it inherits the decision variables from the initial scheduling solution where the fixed operations satisfy the third constraint. The value is used to generate the second assignment model. The calculation of the initial value of the start time variable can be expressed as: ; in, express The The first class of tasks Initial values ​​for the start time variable of each training operation; This represents the set of preceding operations in the execution order of this operation; and These represent the data processing time and data transmission time of the preceding operations, respectively, with the ratio information provided by... The ratio reflects this.

[0109] In this embodiment, by deriving the initial values ​​of decision variables based on the initial scheduling solution, a high-quality starting point is provided for the mixed integer programming model, accelerating the solution convergence process. In the scale reduction scenario, by using operation subset mapping and inheriting fixed operation values, the effectiveness and consistency of the model assignment after reduction are ensured, and effective solutions to large-scale problems are achieved.

[0110] In one embodiment, the step of "scaling down the mixed-integer programming model after assignment to obtain a scaled-down mixed-integer programming model" can be further refined and may include the following steps: Perform at least one of the first scale reduction operation, the second scale reduction operation, and the third scale reduction operation on the assigned mixed integer programming model in sequence until the number of constraints of the reduced mixed integer programming model is less than or equal to the preset scale threshold, and obtain the reduced mixed integer programming model. The first scale reduction operation is: fixing the training operations belonging to the first type of execution phase in the mixed integer programming model after assignment; the second scale reduction operation is: selecting some tasks from the initial scheduling solution to reconstruct the mixed integer programming model after assignment, so that there is at least one training operation belonging to the first type of execution phase on each training device; the third scale reduction operation is: solving only the training operations belonging to the second type of execution phase with a preset proportion. The first type of execution phase and the second type of execution phase are different execution phases.

[0111] Specifically, considering that the solution complexity of mixed integer programming models increases with the number of constraints, large-scale problems may lead to excessively long solution times or failure to converge, this embodiment proposes a scheme to dynamically adjust the model size through a hierarchical reduction strategy.

[0112] To effectively control computational resource consumption and time overhead during the solution process, at least one of the following three scaling operations needs to be sequentially executed on the assigned mixed-integer programming model until the number of constraints in the scaled mixed-integer programming model is less than or equal to a preset scaling threshold, thus obtaining the scaled mixed-integer programming model. "Sequential execution" refers to applying the scaling operations step by step in a predetermined order, re-evaluating the number of constraints after each operation; "until the number of constraints in the scaled mixed-integer programming model is less than or equal to the preset scaling threshold" means monitoring the number of constraints in real time and terminating the scaling process when it reaches or falls below the threshold.

[0113] The first scaling operation specifically manifests as the training operations belonging to the first type of execution phase in the mixed-integer programming model after fixing the assigned values. Its purpose is to reduce the number of decision variables by fixing the training operations in key phases, thereby reducing the model's solution complexity.

[0114] The second scaling operation involves selecting a subset of tasks from the initial scheduling solution to reconstruct the assigned mixed-integer programming model, ensuring that each training device has at least one training operation belonging to the first type of execution phase. Its purpose is to reduce the problem size by decreasing the number of tasks, while maintaining the feasibility of the scheduling scheme.

[0115] The third scale reduction operation specifically involves solving only a predetermined proportion of training operations belonging to the second type of execution phase. Its purpose is to further reduce the scope of optimizable operations, focusing on optimizing training operations in non-critical phases and improving solution efficiency.

[0116] Regarding this step, in some possible implementations, a first scale reduction operation can be performed on the assigned mixed-integer programming model to fix all training operations belonging to the first type of execution phase. If the number of constraints is still greater than the preset scale threshold, a second scale reduction operation is performed to select some tasks from the initial scheduling solution to reconstruct the assigned mixed-integer programming model, ensuring that there is at least one training operation belonging to the first type of execution phase on each training device. If the number of constraints is still greater than the preset scale threshold, a third scale reduction operation is performed to solve only a preset proportion of training operations belonging to the second type of execution phase, until the number of constraints is less than or equal to the preset scale threshold.

[0117] In this embodiment, a hierarchical reduction mechanism is formed by sequentially executing at least one of the first, second, and third scale reduction operations to dynamically adjust the assigned mixed-integer programming model. This mechanism reduces decision variables by fixing the training operations belonging to the first execution phase, reduces the problem size by selecting a subset of tasks from the initial scheduling solution to reconstruct the assigned mixed-integer programming model, and limits the solution scope by solving only a predetermined proportion of the training operations belonging to the second execution phase, thus gradually reducing the number of constraints in the mixed-integer programming model. When the number of constraints drops to or below a predetermined scale threshold, the reduced mixed-integer programming model is obtained. This process reduces the problem size while retaining the key scheduling logic in the initial scheduling solution, allowing for the solution of large-scale pipelined parallel training scheduling problems within limited computing resources.

[0118] In one embodiment, the above step "solving the second assignment model and scaling it to obtain the second scheduling solution" can be further refined and may include the following steps: Solve the second assignment model to obtain the reduced scheduling solution; Extract the first scheduling information of each training operation belonging to the first type of execution phase from the reduced scheduling solution; Extract the second scheduling information of each training operation belonging to the first type of execution phase from the initial scheduling solution; From the reduced scheduling solution, extract the third scheduling information of each training operation after the first type of execution stage, and update the task index of each training operation in the third scheduling information based on the preset index offset to obtain the updated third scheduling information. Based on the first scheduling information, the second scheduling information, and the updated third scheduling information, a second scheduling solution is generated.

[0119] Specifically, considering that the scheduling solution obtained after scaling down needs to match the original problem size to maintain scheduling consistency, this embodiment adopts a scheme to expand the size through stage partitioning and index offset.

[0120] First, the second assignment model needs to be solved to obtain the reduced scheduling solution.

[0121] Regarding this step, in some possible implementations, a mixed-integer programming solver can be invoked to optimize the second assignment model and generate a reduced scheduling solution, which contains the start time and execution order information of each training operation under the reduced problem size.

[0122] Furthermore, it is necessary to extract the first scheduling information of each training operation belonging to the first type of execution stage from the reduced scheduling solution.

[0123] Regarding this step, in some possible implementations, the reduced scheduling solution can be traversed to filter out the training operations whose execution order is before the first type of execution stage. The start time, operation type and device allocation information of these training operations can be extracted as the first scheduling information. The first scheduling information is used to characterize the operation scheduling details in the initial pipeline stage.

[0124] Furthermore, it is necessary to extract the second scheduling information of each training operation belonging to the first type of execution phase from the initial scheduling solution.

[0125] Regarding this step, in some possible implementations, the initial scheduling solution can be parsed, all training operations belonging to the first type of execution phase can be identified, and the execution sequence and resource usage data of these training operations can be obtained as the second scheduling information. The second scheduling information is used to describe the operation configuration of the key overlapping phases.

[0126] Furthermore, it is necessary to extract the third scheduling information for each training operation belonging to the first type of execution phase from the reduced scheduling solution. The task indexes of each training operation in the third scheduling information are then updated based on a preset index offset to obtain the updated third scheduling information. Here, the preset index offset refers to the value used to adjust the task indexes of the training operations to match the original problem size; this value is calculated based on the difference in the number of micro-batches before and after reduction. The task index of each training operation refers to the unique identifier of the training operation within the micro-batch, used to locate the operation's position in the scheduling sequence.

[0127] Regarding this step, in some possible implementations, all training operations after the first type of execution phase can be extracted from the reduced scheduling solution to form third scheduling information; then, the task index of each training operation in the third scheduling information is incrementally updated according to the preset index offset to generate updated third scheduling information, so as to ensure that the index is consistent with the original problem size.

[0128] For example, the preset index offset calculation formula can be expressed as: ; in, Indicates the preset index offset; This represents the total number of micro-batches within the original problem size; This represents the total number of micro-batches under the reduced problem size. After the task index is updated, the new index for the training operation is the original index plus... To align with the original scheduling sequence.

[0129] Finally, based on the first scheduling information, the second scheduling information, and the updated third scheduling information, a second scheduling solution is generated.

[0130] Regarding this step, in some possible implementations, the first scheduling information, the second scheduling information, and the updated third scheduling information can be merged in chronological order to construct a complete scheduling sequence, and this sequence can be converted into executable scheduling instructions as the second scheduling solution. The second scheduling solution covers all training operations in the entire pipeline parallel training process.

[0131] In this embodiment, first scheduling information, second scheduling information, and third scheduling information are extracted through stage division. The initial stage logic of the reduced scheduling solution, the key overlapping stage configuration of the initial scheduling solution, and the subsequent stage structure of the reduced scheduling solution are retained respectively. The task index in the third scheduling information is numerically adjusted by combining a preset index offset so that the updated third scheduling information is aligned with the original problem size in the micro-batch sequence. Finally, the second scheduling solution is generated by merging the three types of scheduling information, which effectively achieves the balance between solution efficiency and scheduling accuracy in large-scale pipeline parallel training scheduling problems.

[0132] In one embodiment, the preset scheduling strategy is the DualPipeV scheduling strategy. The first type of execution phase includes an overlapping phase, and the second type of execution phase includes a reverse-overlapping phase, an interleaved reverse phase, and a cooling phase. The overlapping phase is used to characterize the phase in which training operations for forward propagation tasks and training operations for backward propagation tasks are executed alternately on the training device to mask the time-consuming data transmission phase. The "one-back-propagation-one-overlapping phase" is used to characterize the phase in which the training operations of a single backpropagation class task and the training operations of a single forward propagation class task are executed concurrently on the training device. The staggered backpropagation phase is used to characterize the phase in which training operations for backpropagation-like tasks are performed in an alternating manner on the training device. The cooling phase is used to characterize the training operations of backpropagation-like tasks executed sequentially on the training device to complete the pipelined parallel training process.

[0133] Specifically, considering the complexity of operation scheduling and the need for communication and computational collaboration during pipelined parallel training, this embodiment proposes a stage-based scheduling scheme based on the DualPipeV scheduling strategy to optimize the interleaved execution of training operations for forward propagation tasks and training operations for backward propagation tasks, thereby masking the time consumption of data transmission.

[0134] The DualPipeV scheduling strategy has multiple execution phases, including: warm-up execution phase, staggered forward propagation execution phase, one-forward-one-weight-one-backward execution phase, overlapping execution phase, one-backward-one-overlapping execution phase, staggered back propagation execution phase, and cooling execution phase.

[0135] The execution phases of the DualPipeV scheduling strategy are specifically characterized as follows: Warm-up execution phase: This phase sequentially executes forward propagation tasks on the training device to gradually start the pipeline; Interleaved forward propagation execution phase: This phase interleaves the execution of forward propagation tasks on the training device; One forward-one-weight-one-backward execution phase: This phase sequentially executes a training operation of one forward propagation task, one weight update task, and one backpropagation task on the training device; Overlapping execution phase: This phase interleaves the execution of forward and backpropagation tasks on the training device to mask data transmission time; One backward-overlapping execution phase: This phase overlaps the execution of a single backpropagation task and a single forward propagation task on the training device; Interleaved backpropagation execution phase: This phase interleaves the execution of backpropagation tasks on the training device; Cool-down execution phase: This phase sequentially executes backpropagation tasks on the training device to complete the pipelined parallel training process.

[0136] The first type of execution phase in this embodiment includes an overlapping phase, and the second type of execution phase includes a reverse-overlap phase, an interleaved reverse phase, and a cooling phase. In combination with the above-mentioned related embodiments, it can be expressed as follows: on the training device, the training operations of forward propagation tasks and backpropagation tasks are executed in an interleaved manner to achieve full-to-full communication masking as the overlapping phase; on the training device, the training operations of a single backpropagation task are executed in an overlapping manner with the training operations of a single forward propagation task, the training operations of backpropagation tasks are executed in an interleaved manner, and the training operations of backpropagation tasks are executed sequentially to complete the training as the reverse-overlap phase, the interleaved reverse phase, and the cooling phase.

[0137] In this embodiment, the training operation scheduling on the training device is optimized by dividing the execution phase based on the DualPipeV scheduling strategy. During the overlapping phase, the training operations of forward propagation tasks and back propagation tasks are executed alternately, which masks the data transmission time and improves the overall efficiency of pipelined parallel training.

[0138] In one embodiment, the DualPipeV scheduling strategy is used as the preset scheduling strategy, and relevant embodiments of the scheduling method for training the above model are comprehensively illustrated.

[0139] First, the training parameters corresponding to the large language model are obtained, including model placement order parameters, modeling granularity parameters, pipeline stage parameters, and micro-batch parameters. Based on the training parameters, the proportion information of training operations of each training device in the pipeline parallel training process corresponding to the large language model is determined. This proportion information is used to characterize the ratio between the data processing time and data transmission time of the corresponding training operation. In this embodiment, the modeling granularity parameter is set to the granularity of attention layer data processing and data transmission operations. The specific proportion information is shown in Table 1: Table 1: Ratio of Data Processing Time to Data Transmission Time in Training Operations Training Operation The ratio between data processing time and data transmission time F_atten (Forward Attention) 0.5 F_comm1 (Forward Communication 1) 0.52 F_mlp (Forward MLP) 0.5 F_comm2 (Forward Communication 2) 0.52 W_atten (weighted attention update) 0.25 W_mlp (Weighted Update MLP) 0.5 B_mlp (Reverse MLP) 0.6 B_comm3 (Reverse Communication 3) 0.45 B_atten (backward attention) 1.0 B_comm4 (Reverse Communication 4) 0.52 Based on the DualPipeV scheduling strategy and the proportion of training operations on each training device, an initial scheduling solution is determined. This process includes: firstly, dividing the training process into seven execution phases according to the DualPipeV strategy: warm-up execution phase, staggered forward propagation execution phase, one-forward-one-weight-one-backward execution phase, overlapping phase, one-backward-one-overlapping execution phase, staggered backpropagation execution phase, and cool-down execution phase. The DualPipeV strategy generates the execution order of each training operation on each training device. Subsequently, based on this execution order and the ratio between the actual data processing time and data transmission time in Table 1, the start time of each training operation is derived through an algorithm, thus forming an initial scheduling solution that includes the execution order and start time.

[0140] Based on the initial scheduling solution, the preset objective function, and the preset set of constraints, a mixed-integer programming model is determined. The preset objective function is used to minimize the total execution time of the target scheduling solution. The preset set of constraints includes: a first constraint (constraining the training operations of forward propagation tasks and backpropagation tasks in the same micro-batch to be executed in a predetermined order), a second constraint (constraining the start time of the training operation of weight update tasks to be no earlier than the end time of the backpropagation task with the same sequence number), a third constraint (constraining the execution time of any two training operations on the same training device to not overlap), a fourth constraint (constraining the memory usage to not exceed the upper limit), and a fifth constraint (constraining the total execution time to be no less than the completion time of any training operation). In this embodiment, the specific memory constraint of the fourth constraint is: F_atten and F_mlp operations consume 1 unit of memory, W_atten and W_mlp operations release 1 unit of memory, and the memory change for other operations is 0. The maximum memory upper limit is set to the peak memory value of the initial scheduling solution.

[0141] A scaling-adaptive solution is performed based on the initial scheduling solution and the mixed-integer programming model. First, it is determined whether the number of constraints in the mixed-integer programming model exceeds a preset scale threshold (set to 1 million in this embodiment). If the number of constraints does not exceed 1 million, the mixed-integer programming model is assigned values ​​based on the initial scheduling solution to obtain the first assigned model. Specifically, according to the execution order of training operations of each training device in the initial scheduling solution and the proportional information in Table 1, the initial value of the start time variable for each training operation is determined, and the decision variable is assigned values ​​accordingly. The first assigned model is solved to obtain the first scheduling solution, which is then determined as the target scheduling solution. If the number of constraints exceeds 1 million, the scale of the assigned mixed-integer programming model is reduced, and a solution strategy combining scale reduction and expansion is adopted. First, the first scale reduction operation is executed, that is, fixing the training operations in the assigned mixed-integer programming model that belong to the overlapping stage (i.e., the first type of execution stage). If the number of constraints after reduction still exceeds the threshold, a second reduction operation is performed. This involves reconstructing the mixed-integer programming model by reducing the number of micro-batches, based on the first reduction operation. During reconstruction, it is ensured that at least one training operation belonging to the overlapping phase exists on each training device. If the number of constraints still exceeds the threshold, a third reduction operation is performed. This involves solving only the training operations belonging to the one-backward-overlapping phase, the staggered backpropagation execution phase, and the cooling execution phase (i.e., the second type of execution phase), based on the execution order and proportion information of the corresponding part in the initial scheduling solution, to obtain the second assigned model. Solving the second assignment model yields a reduced scheduling solution, and scaling is then performed to obtain a second scheduling solution. This process includes: extracting first scheduling information from the reduced scheduling solution for each training operation before the overlapping stage; extracting second scheduling information from the initial scheduling solution for each training operation in the overlapping stage; extracting third scheduling information from the reduced scheduling solution for each training operation after the overlapping stage; updating the task index of each training operation in the third scheduling information based on a preset index offset (the difference between the original total number of micro-batches and the reduced total number of micro-batches) to obtain updated third scheduling information; generating a second scheduling solution based on the first scheduling information, the second scheduling information, and the updated third scheduling information, and determining it as the target scheduling solution. The pipeline parallel training process is then scheduled based on the obtained target scheduling solution. Test results on a real cluster are shown in Table 2, where the task size is expressed as (number of pipeline stages, number of micro-batches): Table 2: Comparison of total execution time for scheduling methods at different scales Task size (number of pipeline stages, number of micro-batches) Total execution time corresponding to the initial scheduling solution (units omitted) Total execution time corresponding to the target scheduling solution (units omitted) Total execution time optimization rate Remark (4,10) 189.12 179.49 5.1% Global Optimum (16,64) 1110.00 1090.11 1.8% (16,120) 1862.64 1842.75 1.07% Derived from 64 micro-batch Tests show that this method optimizes scheduling solution generation by introducing information on the ratio of real data processing time to data transmission time, and achieves global scheduling optimization under memory constraints by utilizing a mixed-integer programming model and a scale-adaptive solution mechanism. For large-scale problems (such as 16 pipeline stages and 120 micro-batches), a 1.07% performance improvement can still be achieved by deriving and expanding a model with 64 micro-batches, effectively overcoming the masking defect of all-to-all communication in the training of hybrid expert models and improving the overall efficiency of pipeline parallel training.

[0142] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device may include: a processor 1301, a communication interface 1302, a memory 1303, and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other via the communication bus 1304. The processor 1301 can call a computer program in the memory 1303 to execute the steps of a model training scheduling method, such as including: Obtain the training parameters corresponding to the large language model, and determine the proportion information of training operations of each training device in the pipeline parallel training process corresponding to the large language model based on the training parameters. The proportion information is used to characterize the ratio between the data processing time and the data transmission time of the corresponding training operation. The initial scheduling solution is determined based on the preset scheduling strategy and the proportion of training operations of each training device. Based on the initial scheduling solution, the preset objective function, and the preset set of constraints, a mixed integer programming model is determined. The target scheduling solution is obtained by adaptively solving the scale based on the initial scheduling solution and the mixed integer programming model. Based on the target scheduling solution, the pipeline parallel training process is scheduled.

[0143] Furthermore, when the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0144] On the other hand, embodiments of this application also provide a non-transitory computer-readable storage medium storing a computer program. The computer program is used to cause a processor to execute the steps of the methods provided in the above embodiments, including, for example: Obtain the training parameters corresponding to the large language model, and determine the proportion information of training operations of each training device in the pipeline parallel training process corresponding to the large language model based on the training parameters. The proportion information is used to characterize the ratio between the data processing time and the data transmission time of the corresponding training operation. The initial scheduling solution is determined based on the preset scheduling strategy and the proportion of training operations of each training device. Based on the initial scheduling solution, the preset objective function, and the preset set of constraints, a mixed integer programming model is determined. The target scheduling solution is obtained by adaptively solving the scale based on the initial scheduling solution and the mixed integer programming model. Based on the target scheduling solution, the pipeline parallel training process is scheduled.

[0145] Non-transitory computer-readable storage media can be any available medium or data storage device that can be accessed by a processor, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0146] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A scheduling method for model training, characterized in that, include: Obtain the training parameters corresponding to the large language model, and determine the proportion information of the training operations of each training device in the pipeline parallel training process corresponding to the large language model based on the training parameters. The proportion information is used to characterize the ratio between the data processing time and the data transmission time of the corresponding training operation. The initial scheduling solution is determined based on the preset scheduling strategy and the proportion of training operations of each training device. Based on the initial scheduling solution, the preset objective function, and the preset set of constraints, a mixed integer programming model is determined; Based on the initial scheduling solution and the mixed integer programming model, an adaptive scaling solution is performed to obtain the target scheduling solution; Based on the target scheduling solution, the pipeline parallel training process is scheduled.

2. The method according to claim 1, characterized in that, The process of obtaining the training parameters corresponding to the large language model includes: Obtain the model placement order parameters, modeling granularity parameters, pipeline stage parameters, and micro-batch parameters corresponding to the large language model; The training parameters corresponding to the large language model are determined based on the model placement order parameters, modeling granularity parameters, pipeline stage parameters, and micro-batch parameters.

3. The method according to claim 1, characterized in that, The step of determining the initial scheduling solution based on the preset scheduling strategy and the proportion information of the training operations of each training device includes: Based on the preset scheduling strategy and the proportion information of training operations of each training device, multiple execution stages are determined; The initial scheduling solution is determined based on the multiple execution phases.

4. The method according to claim 1, characterized in that, The step of determining the mixed-integer programming model based on the initial scheduling solution, the preset objective function, and the preset set of constraints includes: Based on the initial scheduling solution, obtain the initialized decision variables; Based on the aforementioned decision variables, preset objective function, and preset set of constraints, a mixed integer programming model is constructed. The preset objective function is used to minimize the total execution time for determining the target scheduling solution.

5. The method according to claim 4, characterized in that, The set of preset constraints includes at least one of the following: The first constraint condition is used to constrain the training operations of forward propagation tasks and backward propagation tasks in the same micro-batch to be executed in a predetermined order by using the part of the decision variable that represents the start time of the training operation. The second constraint is used to constrain, through the part of the decision variable that represents the start time of the training operation, that the start time of any training operation of the weight update class task in the same micro-batch is not earlier than the end time of the backpropagation class task with the same operation number. The third constraint is used to constrain the execution times of any two training operations on the same training device to not overlap, by using the part of the decision variable that represents the execution order of the training operations. The fourth constraint is used to ensure that when performing any training operation on any of the training devices, the sum of the video memory consumed by the training operation already performed on the training device and the video memory required for the current training operation does not exceed a preset video memory limit. The fifth constraint is to ensure that the total execution time minimized by the preset objective function is not less than the completion time of any of the training operations.

6. The method according to claim 5, characterized in that, The first constraint is further used to constrain the following time relationship using the portion of the decision variable that represents the start time of the training operation: For the same forward propagation task or the same backward propagation task in the same micro-batch, the start time of the next training operation shall not be earlier than the sum of the execution time and communication time of the previous training operation. For the same micro-batch, the start time of the first training operation of a backpropagation task is no earlier than the sum of the execution time and communication time of the last training operation of a forward propagation task.

7. The method according to claim 1, characterized in that, The step of adaptively solving the target scheduling solution based on the initial scheduling solution and the mixed integer programming model includes: If the number of constraints in the mixed integer programming model does not exceed a preset scale threshold, then the mixed integer programming model is assigned a first assigned model based on the initial scheduling solution, the first assigned model is solved to obtain a first scheduling solution, and the first scheduling solution is determined as the target scheduling solution. If the number of constraints in the mixed integer programming model exceeds a preset size threshold, the size of the assigned mixed integer programming model is reduced to obtain a reduced mixed integer programming model. Based on the initial scheduling solution, the reduced mixed integer programming model is assigned a second assigned model. The second assigned model is solved and its size is expanded to obtain a second scheduling solution, and the second scheduling solution is determined as the target scheduling solution.

8. The method according to claim 7, characterized in that, The process of assigning values ​​to the mixed-integer programming model based on the initial scheduling solution to obtain the first assigned model includes: Based on the proportion information of the training operations of each training device and the execution order of the training operations of each training device in the initial scheduling solution, the initial value of the start time variable of the training operation of each training device is determined. Based on the initial value of the start time variable of the training operation of each training device, the decision variables in the mixed integer programming model are assigned values ​​to obtain the first assignment model; The process of assigning values ​​to the reduced mixed-integer programming model based on the initial scheduling solution to obtain the second assigned model includes: Based on the proportion information of the training operations of each training device and the execution order of the part of the initial scheduling solution corresponding to the reduced mixed integer programming model, the initial value of the start time variable of the training operation of each training device is determined. Based on the initial value of the start time variable of the training operation of each training device, the decision variables in the reduced mixed integer programming model are assigned values ​​to obtain the second assignment model.

9. The method according to claim 7, characterized in that, The process of scaling down the assigned mixed-integer programming model to obtain a scaled-down mixed-integer programming model includes: The mixed integer programming model after assignment is subjected to at least one of the first scale reduction operation, the second scale reduction operation and the third scale reduction operation in sequence until the number of constraints of the reduced mixed integer programming model is less than or equal to the preset scale threshold, so as to obtain the reduced mixed integer programming model. The first scale reduction operation is: fixing the training operations belonging to the first type of execution stage in the assigned mixed integer programming model; the second scale reduction operation is: selecting a portion of tasks from the initial scheduling solution to reconstruct the assigned mixed integer programming model, so that there is at least one training operation belonging to the first type of execution stage on each training device; the third scale reduction operation is: solving only a preset proportion of the training operations belonging to the second type of execution stage. The first type of execution phase and the second type of execution phase are different execution phases.

10. The method according to claim 9, characterized in that, The process of solving the second assignment model and scaling it to obtain the second scheduling solution includes: Solving the second assignment model yields the reduced scheduling solution; From the reduced scheduling solution, extract the first scheduling information of each training operation belonging to the first type of execution stage before the first type of execution stage; From the initial scheduling solution, extract the second scheduling information for each training operation belonging to the first type of execution phase; From the reduced scheduling solution, extract the third scheduling information of each training operation after the first type of execution stage, and update the task index of each training operation in the third scheduling information based on the preset index offset to obtain the updated third scheduling information. A second scheduling solution is generated based on the first scheduling information, the second scheduling information, and the updated third scheduling information.

11. The method according to claim 9 or 10, characterized in that, The preset scheduling strategy is the DualPipeV scheduling strategy. The first type of execution phase includes an overlapping phase, and the second type of execution phase includes a reverse-overlapping phase, an interleaved reverse phase, and a cooling phase. The overlapping phase is used to characterize the phase in which training operations of forward propagation tasks and back propagation tasks are executed alternately on the training device to mask the time-consuming data transmission phase. The aforementioned back-and-overlap phase is used to characterize the phase in which the training operation of a single backpropagation class task and the training operation of a single forward propagation class task are executed overlappingly on the training device. The interleaved reverse phase is used to characterize the phase in which training operations of backpropagation-type tasks are executed interleaved on the training device; The cooling phase is used to characterize the phase in which training operations of backpropagation-type tasks are executed sequentially on the training device to complete the pipelined parallel training process.

12. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the scheduling method for model training according to any one of claims 1 to 11.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the scheduling method for model training as described in any one of claims 1 to 11.

Images