Model training scheduling strategy generation method and device and scheduling strategy application method and device

Abstract

The invention provides a model training scheduling strategy generation method, a model is divided into a plurality of stages according to layers, each stage is deployed on different computing equipment, and training data flows through each stage in a micro-batch form. According to the method, firstly, configuration parameters of model training are obtained, and the model training is executed by adopting a pipeline parallel 1F1B strategy; and secondly, performing video memory occupation prediction of the computing equipment according to the configuration parameters. And under the condition that the video memory occupation prediction result exceeds a video memory preset threshold value, modeling is performed on pipeline parallelism. Modeling comprises the step of establishing a microbatch dependency constraint, a video memory window constraint and an activation value transfer number constraint of pipeline parallelism. And finally, generating a new scheduling strategy based on modeling solution. According to the scheme, the problems of load imbalance and key path extension in an assembly line can be solved by combining a global scheduling perspective and an inter-stage coordination mechanism, and the efficiency of an assembly line system is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for generating a model training scheduling strategy, and a method and apparatus for applying a scheduling strategy. Background Technology

[0002] With the success of large language models based on the Transformer architecture across multiple domains, model size and context length are constantly increasing, placing extremely high demands on training hardware resources. To support the training of large models, methods such as data parallelism, tensor parallelism, and pipeline parallelism are commonly used. Pipeline parallelism divides the model into multiple parts, and pipeline bubbles can arise due to data dependencies. While the 1F1B scheduling method can minimize these bubbles, it suffers from uneven memory usage during training. Existing optimization methods either rely solely on recomputation methods to reduce memory usage by adding extra computation, or solely on activation value transfer methods to reduce memory usage by increasing communication. Neither of these methods achieves a balance between computational overhead and memory consumption, nor can they effectively schedule hardware resources to achieve better throughput. Summary of the Invention

[0003] This invention describes a method and apparatus for generating pipelined parallel training scheduling strategies, and a method and apparatus for applying scheduling strategies, which can solve the above-mentioned technical problems.

[0004] According to the first aspect, a method for generating a model training scheduling strategy is provided. The model is divided into multiple stages by layers, with each stage deployed on a different computing device. Training data flows through each stage in micro-batches. In this method, firstly, configuration parameters for model training are obtained. The model training adopts a first scheduling strategy, which is a pipelined parallel 1F1B scheduling strategy. Secondly, the memory usage of the computing device is predicted based on the configuration parameters. If the memory usage prediction result exceeds the memory usage threshold, pipelined parallelism is modeled. This modeling includes establishing micro-batch dependency constraints, memory window constraints, and activation value transition quantity constraints for pipelined parallelism. Finally, a second scheduling strategy is generated based on the modeling results.

[0005] In some embodiments, the configuration parameters include model configuration parameters, computing device cluster configuration parameters, and pipeline parallelism configuration parameters.

[0006] In some embodiments, the pipeline is divided into a preheating state, a stable training state, and a cooling state. In the preheating state, the micro-batch dependency constraint is configured as follows: within the same stage, the start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of the previous micro-batch; across different stages, the start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of the same micro-batch in the previous stage. In a stable training state, the micro-batch dependency constraint is configured as follows: In different alternating cycles, the start time of the forward propagation of the current micro-batch is later than the end time of the backward propagation of the current micro-batch in the previous cycle; where alternating cycles refer to a process of performing one forward propagation and one backward propagation in one phase; the start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of the micro-batch in the previous phase; the start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of the current micro-batch in the previous cycle; the start time of the backward propagation of the current micro-batch is later than the end time of the backward propagation of the micro-batch in the next phase; in the same alternating cycle, the start time of the backward propagation of the current micro-batch is later than the end time of the forward propagation of the current micro-batch in the current cycle. In the cooled state, the micro-batch dependency constraint is configured such that, in different stages, the start time of the backpropagation of the micro-batch in this stage is later than the end time of the backpropagation of the micro-batch in the next stage; and the start time of the backpropagation of the micro-batch in this stage is later than the end time of the backpropagation of the micro-batch in this stage in the previous loop.

[0007] In some embodiments, the memory window constraint includes: within each window, the number of cached micro-batches does not exceed the maximum number of micro-batches that can be stored in the current stage.

[0008] In some embodiments, the activation value transfer quantity constraint includes: the total number of activation value transfers within any window does not exceed the capacity that the current activation value sender cannot store, nor does it exceed the capacity that the current activation value receiver can still store.

[0009] According to the second aspect, a scheduling strategy application method is provided. In this method, firstly, configuration parameters for model training are obtained. The model training employs a first scheduling strategy, which is a pipelined parallel 1F1B scheduling strategy. Secondly, memory usage prediction of the computing device is performed based on the configuration parameters. If the predicted memory usage exceeds a memory usage threshold, a second scheduling strategy is obtained. This second scheduling strategy is based on the pipelined parallel modeling results, where modeling includes establishing micro-batch dependency constraints, memory window constraints, and activation value transfer quantity constraints for pipelined parallelism. Then, the second scheduling strategy is used to schedule model training. The scheduling objects are micro-batches and / or activation values. If the predicted memory usage is less than or equal to the memory usage threshold, the first scheduling strategy continues to be executed.

[0010] In some embodiments, the configuration parameters include model configuration parameters, computing device cluster configuration parameters, and pipeline parallelism configuration parameters.

[0011] In some embodiments, the pipeline is divided into a preheating state, a stable training state, and a cooling state. In the preheating state, the micro-batch dependency constraint is configured as follows: within the same stage, the start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of the previous micro-batch; across different stages, the start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of the same micro-batch in the previous stage. In a stable training state, the micro-batch dependency constraint is configured as follows: In different alternating cycles, the start time of the forward propagation of the current micro-batch is later than the end time of the backward propagation of the current micro-batch in the previous cycle; where alternating cycles refer to a process of performing one forward propagation and one backward propagation in one phase; the start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of the micro-batch in the previous phase; the start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of the current micro-batch in the previous cycle; the start time of the backward propagation of the current micro-batch is later than the end time of the backward propagation of the micro-batch in the next phase; in the same alternating cycle, the start time of the backward propagation of the current micro-batch is later than the end time of the forward propagation of the current micro-batch in the current cycle. In the cooled state, the micro-batch dependency constraint is configured such that, in different stages, the start time of the backpropagation of the micro-batch in this stage is later than the end time of the backpropagation of the micro-batch in the next stage; and the start time of the backpropagation of the micro-batch in this stage is later than the end time of the backpropagation of the micro-batch in this stage in the previous loop.

[0012] In some embodiments, the memory window constraint includes: within each window, the number of cached micro-batches does not exceed the maximum number of micro-batches that can be stored in the current stage.

[0013] In some embodiments, the activation value transfer quantity constraint includes: the total number of activation value transfers within any window does not exceed the capacity that the current activation value sender cannot store, nor does it exceed the capacity that the current activation value receiver can still store.

[0014] This application predicts the memory usage of computing devices. If the predicted memory usage exceeds a preset threshold, pipeline parallelism is modeled. Micro-batch dependency constraints, memory window constraints, and activation value transfer quantity constraints are also set for the model. Finally, the optimal pipeline parallel scheduling strategy is solved. This solution combines a global scheduling perspective and inter-stage coordination mechanisms to solve the problems of load imbalance and critical path extension in the pipeline, thereby improving the efficiency of the pipeline system. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This diagram illustrates a micro-batch active window for various stages of management provided in this specification.

[0017] Figure 2 This diagram illustrates a memory-constrained micro-batch active window provided in this specification.

[0018] Figure 3 This diagram illustrates a flowchart of a model training scheduling strategy generation method provided in an embodiment of this specification.

[0019] Figure 4 This document shows a schematic flowchart of a scheduling strategy application method provided in an embodiment of this specification.

[0020] Figure 5 This diagram illustrates the structure of a model training scheduling strategy generation device provided in an embodiment of this specification.

[0021] Figure 6 This document shows a schematic diagram of the structure of a scheduling strategy application device provided in an embodiment of this specification. Detailed Implementation

[0022] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.

[0024] In the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0025] In the description of the embodiments of this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more.

[0026] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0027] To facilitate understanding of the solutions provided in the embodiments of this application, a brief introduction to some of the terms involved in this solution will be given first.

[0028] Pipeline parallelism is a distributed parallel computing technique used in deep learning model training, primarily to address the problem that ultra-large-scale neural networks cannot be accommodated on a single device. It achieves efficient training by splitting the model itself layer by layer across multiple computing devices (such as GPUs).

[0029] The pipelined parallel 1F1B strategy is mainly used to efficiently train ultra-large-scale models in multi-computing device environments. Its core idea is to split the model into multiple stages, and then further divide the training batches into multiple micro-batches. By interleaving forward and backward propagation, the idle time of each stage can be minimized.

[0030] Micro-batch: In distributed training of deep learning, it is the smallest indivisible unit of computation scheduling. It is a subset of data obtained by further partitioning a macro-training batch along the data dimension.

[0031] In large-scale language model training, hardware resources (such as GPU memory) often become a key factor limiting training efficiency. Especially when adopting a pipelined parallel strategy, how to design the optimal micro-batch scheduling strategy for different model sizes and parallel configurations under the constraint of limited hardware resources is an important and challenging problem.

[0032] To systematically characterize the dynamic changes in activation values ​​during pipeline scheduling, this application introduces the concept of a "Micro-Batch Active Window," which represents the set of micro-batches maintained by a specific stage at a given moment. In 1F1B scheduling, the lifecycle of a micro-batch begins with entry into the first stage of the pipeline, proceeds forward propagation sequentially to the next stage, and ends at the last stage, storing the intermediate activation values ​​generated during the forward computation. It then re-enters from the last stage, proceeds backward propagation sequentially to the previous stage, consuming the intermediate activation values ​​stored in the corresponding stage, until it reaches the first stage. Due to the forward-backward flow characteristics, the number of micro-batch activation values ​​stored on different stages varies, and the earlier the pipeline stage, the more micro-batches are cached.

[0033] Figure 1 A schematic diagram of a micro-batch active window for stage management is shown. Figure 1 This describes the set of eight micro-batches that need to be processed in a four-stage pipelined parallel training. The rectangles in the diagram correspond to the set of micro-batches currently maintained by each stage at a given time, i.e., the active micro-batch window. During the pipelined parallel training process, the stage performs forward propagation on the micro-batches at the right end of the window, saving their generated activation values ​​and adding them to the window. Next, the stage performs backward propagation on the micro-batches at the left end of the window, releasing their activation values ​​and popping them out of the window. This process is repeated. During the stable training phase of the pipeline, the active micro-batch set maintained by the stage can be seen as its window sliding to the right along the micro-batch dimension. The changes in the activation values ​​stored by the stage follow a "generate-release-generate" pattern, maintaining a stable level.

[0034] It's worth noting that because the number of Micro-Batches executed consecutively during the pipeline warm-up phase varies across different Stages, the size of the active Micro-Batch window maintained by each Stage also differs. Specifically, the maximum number of active Micro-Batches that the i-th Stage needs to maintain is Pi, and the corresponding window length is also Pi. However, in environments with limited video memory, the video memory window is insufficient to accommodate the activation values ​​generated by a maximum of P Micro-Batches. Therefore, it is necessary to selectively discard the activation values ​​of Micro-Batches during runtime to ensure that, at any given moment, the number of cached Micro-Batches meets the size of the video memory window.

[0035] Figure 2 This diagram illustrates a micro-batch active window with limited video memory. Figure 2 As shown, for a Stage with a Micro-Batch window of 4 and a Video Memory window of only 2, after caching 2 Micro-Batches, the activation value of the 3rd Micro-Batch can be transferred to a Stage with a Video Memory window larger than the Micro-Batch window, and the activation value can be recalculated for the 4th Micro-Batch.

[0036] Based on the theoretical analysis of the Micro-Batch active window described above, this application derives the principle for selecting activation value recalculation or activation value transfer in Micro-Batch: throughout the entire execution cycle of pipelined parallel training, the Micro-Batch window at any given time should ensure that it contains at least m Micro-Batches for recalculation and n Micro-Batches for activation value transfer. This constraint guarantees that the peak memory usage of a Stage is always limited to the memory capacity required by Pimn Micro-Batches, where i is the Stage number. In other words, this is a sliding window constraint problem: that is, it is necessary to select appropriate positions in the micro-batch sequence for recalculation or activation value transfer, so that any sliding window of length 𝑃−𝑖 contains at least m recalculation points and n transfer points.

[0037] Based on the above theoretical analysis, this application designs a memory-aware decision-making mechanism to achieve micro-batch level scheduling control. This mechanism determines the recomputation and activation value transfer strategy by dynamically maintaining the real-time memory occupancy status of the Stage and combining it with the memory requirements of a single micro-batch.

[0038] To accurately assess the cache limit of each Stage on a Micro-Batch in pipelined parallelism, this application proposes a memory usage prediction model. This model dynamically analyzes and predicts the memory usage of each computing device, including both static and dynamic memory usage, based on model architecture parameters, cluster configuration, and distributed training configuration. Using this predictor, the maximum cache capacity within the Micro-Batch active window can be accurately estimated, and decisions can be made accordingly regarding whether activation value recalculation or activation value transfer is necessary.

[0039] During model training, the memory usage of computing devices can be divided into two components: static memory usage and dynamic memory usage. Static memory mainly consists of fixed overhead such as model parameters, model gradients, and optimizer states, and its size depends on the model structure and the number of parameters deployed on the computing device. Dynamic memory usage is not only related to the model architecture but also to pipeline parallel configuration parameters, such as the sequence length during training, the micro-batch size, and the number of micro-batches active simultaneously in the pipeline parallel system.

[0040] For a typical large language model, each component in its model structure corresponds to a specific number of parameters and intermediate activation values. To systematically analyze the memory usage characteristics of large language models, we first identify the key parameter variables of the model: model structure parameters include hidden layer dimension h, number of Transformer blocks L, number of force heads a, and number of small v; training configuration parameters include sequence length s and micro-batch size mb N; distributed training parameters include tensor parallelism 𝑡 and pipeline parallelism P.

[0041] First is the Embedding layer. In the large language model architecture, the Embedding layer is responsible for mapping discrete input tokens to a continuous vector space. It has v×h parameters and an activation value size of 2×s×mb×v×h.

[0042] Next is the attention layer. The attention layer has four weight matrices, Q, K, V and O, each of size H×H, so the number of parameters is 4×H×H. The storage of its activation values ​​comes from the following computation modules: (1) the shared input of Q, K and V, 2×s×mb×h; (2) QKT needs to store Q and K at the same time, occupying 4×s×mb×h space; (3) the SoftMax operation needs to store 2×a×s 2 The output size requires an a×s Dropout mask. 2 ×mb interval; (4) Finally, in the calculation of attention applied to V, it is necessary to save the Dropout output 2×a×s 2 The values ​​are 2×s×mb×h and V. However, when applying the Flash Attention technique, it does not store the complete attention score matrix and SoftMax output. During backpropagation, these intermediate results are obtained through recomputation. By sacrificing a small amount of computational overhead, the O(s) value in the attention mechanism is significantly reduced. 2 The memory requirements are at the level of 4s × mb × h. Therefore, the total activation value storage requirement for the attention layer is 4 × s × mb × h.

[0043] Finally, there's the MLP layer. The MLP layer typically uses a two-layer feedforward neural network structure, containing a first linear transformation layer that expands the hidden layer dimension from h to 4h, with a parameter matrix of size h×4h. The second linear transformation layer reduces the dimension back to h from 4h. Therefore, the parameter matrix size of the MLP layer is 8h. 2 The total storage space occupied by the activated values ​​is 19 × s × mb × h.

[0044] In summary, the total GPU memory required for training the aforementioned large model is:

[0045]

[0046] When tensor parallelism is introduced, the memory usage on each computing device is:

[0047]

[0048] When pipelined parallelism is introduced, the memory usage of the computing device on the i-th stage is:

[0049]

[0050] Therefore, the following memory constraints exist during pipeline scheduling:

[0051]

[0052] in, This represents the maximum number of micro-batches that this stage can buffer. Solving this constraint yields... The maximum value provides a parameter basis for subsequent system modeling.

[0053] The micro-batch active window theory in pipelined parallelism reveals the dynamic changes in GPU memory at each stage during training. Under memory constraints, it's necessary to select a specific number of micro-batches within a window to perform activation recalculation or activation transfer operations. However, this is a complex system optimization problem. While window theory limits the number of decisions at the stage level—simply ensuring that the number of cached micro-batches remains within the memory window at any given time—from the perspective of the entire pipeline system, the scheduling strategy made by a stage can affect the startup time of adjacent stages, causing the current stage to become a bottleneck, increasing the length of the critical path, and reducing the efficiency of the pipeline system.

[0054] To address the aforementioned issues, this application further proposes a pipeline scheduling optimization framework based on integer programming. This framework primarily focuses on two types of key dependency constraints in pipeline scheduling: first, dependency constraints between different micro-batches within the same stage; and second, dependency constraints between the same micro-batches in adjacent stages.

[0055] Specifically, the first type of constraint manifests as the strict temporal order of the computation sequence. For example, for any micro-batch n and m in any stage, if n enters the system before m, then the forward and backward propagation operations of micro-batch n must be completed strictly before the corresponding operations of micro-batch m. The second type of constraint manifests as the spatial dependency of the computation process. For example, each micro-batch follows a strict forward propagation order (e.g., from Stage 0 to Stage P-1, where P refers to the total number of stages) and a backward propagation order (from Stage P-1 to Stage 0) between stages.

[0056] To facilitate the establishment of constraints, the model parameters and decision variables in the pipeline parallel mode are first formally defined.

[0057] Specifically, for one iteration of pipeline parallelism, we can let P be the pipeline parallelism dimension, that is, the number of stages; and let rank represent the number of the current stage.

[0058] Let M be the number of micro-batches in the system; let Microbatchid represent the current micro-batch number;

[0059] It can make For the operation selection to be performed on the j-th micro-batch on the i-th stage, when k=0, the operation is marked as normal forward propagation, when k=1, the operation is marked as activation value transfer, and when k=2, the operation is marked as activation value recalculation.

[0060] It can make This represents the time when the j-th micro-batch operation begins in the i-th stage. When k=0, it represents the start time of forward propagation, and when k=1, it represents the start time of backward propagation.

[0061] CostTransferForward and CostTransferBackward can be represented as the costs of forward and backward propagation under the activation value transfer strategy.

[0062] CostTransferForward and CostTransferBackward can be represented as the costs of forward and backward propagation under the activation value recalculation strategy.

[0063] CostForward and CostBackward can be used to represent the costs of the default forward and backward propagation.

[0064] It can make The cost of forward propagation (k=0) or backward propagation (k=1) of the j-th microbatch on the i-th stage.

[0065] For simplicity, the forward propagation time can be set to 1 unit of time by default in subsequent calculations, the back propagation time can be set to 2 units of time, and the back propagation time including recalculation can be set to 3 units of time.

[0066] Based on the above definition, an integer programming model can be constructed. Taking the time point when the first stage completes the backpropagation of its last micro-batch as the optimization objective, micro-batch dependency constraints are set according to the characteristics of the three stages of the 1F1B scheduling strategy. The objective function of this integer programming model can be expressed by formula (1):

[0067] (1)

[0068] In the formula, it is possible to... Set the initial time to 0, that is... .

[0069] The parallel pipeline process has three states from start to finish: warm-up state, stable training state, and cooling state.

[0070] The first state is the preheating state of pipeline parallelism, where dependency constraints on micro-batches can be established. In the preheating state, the Stage at the rank position will continuously process the forward propagation of P-rank-1 micro-batches. As shown in Equation (2), the start time of the forward propagation of the j-th micro-batch on the i-th Stage is always later than the end time of the (j-1)-th forward propagation on the i-th Stage. As shown in Equation (3), the start time of the forward propagation of the j-th micro-batch on the i-th Stage is also always later than the end time of the j-th forward propagation on the (i-1)-th Stage. That is, in the same stage, the start time of the forward propagation of this micro-batch is later than the end time of the forward propagation of the previous micro-batch. In different stages, the start time of the forward propagation of the micro-batch in this stage is later than the end time of the forward propagation of the same micro-batch in the previous stage.

[0071] (2)

[0072] (3)

[0073] The second state is the stable training state for pipelined parallelism, where dependency constraints on micro-batches can be established. In this stable training state, the stage at the rank position alternately processes the forward and backward propagation of different micro-batches. The total loop... A stage alternates once, meaning that a stage performs one forward propagation and one backward propagation. In the formula, j can be let be the iteration variable. Let this be the batch number used for forward propagation in the j-th iteration. This is the micro-batch number used for backpropagation in the j-th iteration. Specifically,

[0074] (4)

[0075] (5)

[0076] (6)

[0077] (7)

[0078] (8)

[0079] Equations (4), (5), and (6) above are dependency constraints set for the forward propagation of micro-batches in a stable training state. As shown in equation (4), the start time of the forward propagation of the micro-batches in this stage is always later than the end time of the backward propagation of the micro-batches in the previous cycle. As shown in equation (5), the start time of the forward propagation in this stage is also always later than the end time of the forward propagation of the micro-batches in the previous stage. As shown in equation (6), the start time of the forward propagation in this stage is always later than the end time of the forward propagation in the previous cycle. That is to say, in different alternating cycles, the start time of the forward propagation of the micro-batches in this stage is later than the end time of the backward propagation of the micro-batches in this stage in the previous cycle. Specifically, alternating cycles refer to the process of performing one forward propagation and one backward propagation in one stage. The start time of the forward propagation of the micro-batches in this stage is later than the end time of the forward propagation of the micro-batches in the previous stage. The start time of the forward propagation of the micro-bats in this stage is later than the end time of the forward propagation of the micro-bats in this stage in the previous cycle.

[0080] Equations (7) and (8) above are dependency constraints set for micro-batches during backpropagation in a stable training state. As shown in equation (7), the start time of backpropagation in this stage is always later than the end time of backpropagation of the micro-batches in the next stage. As shown in equation (8), the start time of backpropagation in this stage is always later than the end time of forward propagation in this round of the loop. In other words, the start time of backpropagation of the micro-batches in this stage is later than the end time of backpropagation of the micro-batches in the next stage. In the same round of alternating loops, the start time of backpropagation of the micro-batches in this stage is later than the end time of forward propagation of the micro-batches in this stage in this round of the loop.

[0081] The third state is the cooling state of pipelined parallelism, where dependency constraints on micro-batches can be established in the stable training state of pipelined parallelism. In this stage, each stage will process the remaining... Backpropagation of micro-batch.

[0082] (9)

[0083] (10)

[0084] In the formula, To process the micro-batch number of the j-th backpropagation, and .

[0085] In formula (9), the start time of backpropagation in this stage is always later than the end time of backpropagation of the micro-batch after the next stage has finished processing it. In formula (10), the start time of backpropagation in this stage is always later than the end time of backpropagation of the micro-batch in the previous loop. That is to say, in different stages, the start time of backpropagation of the micro-batch in this stage is later than the end time of backpropagation of the micro-batch after the next stage has finished processing it; the start time of backpropagation of the micro-batch in this stage is later than the end time of backpropagation of the micro-batch in this stage in the previous loop.

[0086] In addition to establishing constraints for the three pipeline parallel states mentioned above, constraints can also be established for the video memory window, as shown in formula (11):

[0087] (11)

[0088] (12)

[0089] (13)

[0090] (14)

[0091] Specifically, continue to refer to Figure 2In a pipelined parallel 1F1B scheduling, each stage needs to manage the activation values ​​of multiple active micro-batches. For the i-th stage, the active micro-batch window size is Pi, representing the number of micro-batches that the stage needs to manage simultaneously at any given time. The window moves along the micro-batch sequence; when the activation value of a micro-batch is released, the window slides forward one step to allocate memory space for the new micro-batch. Simultaneously, the number of cached micro-batches within each window should not exceed BufferCapcity. BufferCapcity can be used to predict the maximum number of micro-batches that the current stage can store using a memory overhead predictor. Micro-batches exceeding BufferCapcity will undergo activation value recalculation or activation value transfer operations. Whether to perform activation value recalculation or activation value transfer depends on the transmission cost and the remaining memory of the maximum number of acceptor stages that can be stored.

[0092] Formula (15) is a constraint on the number of activation values ​​transferred within any window. Specifically, the total number of activation values ​​transferred should not exceed the total amount that the current Ejector Stage cannot store, nor should it exceed the remaining storage capacity of the Accept Stage. Specifically, the total amount that the current Ejector Stage cannot store = the total number of activation values ​​generated within the current window - the maximum capacity that the Ejector Stage itself can retain (e.g., the maximum video memory capacity of the GPU). The remaining storage capacity of the Accept Stage = the total capacity of the Accept Stage - the space already occupied by the Accept Stage.

[0093] The windows and windows' can be used to represent the active micro-batch windows of the Ejector Stage and the Acceptor Stage, respectively, while the buffer and buffer' represent the storage capacity of the hardware of the Ejector Stage and the Acceptor Stage, respectively.

[0094] (15)

[0095] Based on the above modeling approach, this application can formalize the pipeline parallel scheduling problem into an integer programming problem, which can be solved using commercial solvers (such as Gurobi) to obtain the globally optimal scheduling strategy. The optimal scheduling strategy can then be provided to the pipeline scheduler for dynamic strategy processing of micro-batches.

[0096] The method provided by this invention models pipeline parallelism and establishes micro-batch dependency constraints, memory window constraints, and activation value transfer quantity constraints for pipeline parallelism. The system can find an optimal scheduling method that makes the scheduling of activation values ​​and / or micro-batches in each stage as smooth as possible, and makes the calculation and communication of activation values ​​overlap as much as possible.

[0097] The above content describes how to model pipeline parallelism and solve for the optimal scheduling strategy. Next, we will talk about how to schedule the optimal strategy.

[0098] This application provides a scheduling framework, OmniPipe, for executing operations with optimal scheduling strategies.

[0099] Based on the above understanding, pipelined parallel scheduling can be divided into three states: warm-up state, stable training state, and cool-down state. In the warm-up state, each pipeline stage receives activation input from the previous stage and sequentially initiates forward computation. In the stable training state, each pipeline stage alternately executes forward and backward propagation of different micro-batches. In the cool-down state, each pipeline stage continues to execute the backward propagation of the remaining micro-batches.

[0100] Before each forward or backward propagation, the scheduling framework obtains an optimized sequence of operations (i.e., the optimal scheduling strategy). For activation value transfer operations, the system will transfer the saved intermediate activation values ​​to the target device in a timely and asynchronous manner; for recalculation operations, the system will perform forward propagation in advance to obtain the necessary activation values.

[0101] The scheduling framework OmniPipe can be developed based on Megatron-LM. In the pipeline scheduling implementation of Megatron-LM, it can be divided into the three states mentioned above: warm-up state, stable training state, and cooling state.

[0102] The first stage is the warm-up phase. In this phase, the pipeline is not yet "full," and each stage sequentially initiates forward computation. First, it receives activation input from the previous stage. The current stage can call the forward computation function to obtain the output activation and then send that activation to the next pipeline stage.

[0103] Secondly, the training state is stabilized. Each stage, upon receiving activations from the previous stage, performs forward computation to generate the current output. It then sends the current forward activations to the next stage and simultaneously receives backpropagation gradients from the subsequent stage. Simultaneously, it selects the earliest activation value saved in the saved queue from the forward computation, performs backpropagation computation to obtain the input gradient, and then sends it to the previous pipeline stage.

[0104] Finally, there's the cooling-off state. After the stable training state ends, some micro-batches in the pipeline still haven't completed backpropagation calculations. At this point, based on the saved input and forward activation, they are sequentially retrieved from the queue, the remaining subsequent backward gradients are received, backpropagation calculations are performed, and the obtained gradients are sent to the previous stage. This ensures that all micro-batches have completed forward and backward calculations, and gradient synchronization / reduction is also finished.

[0105] This application also provides an algorithm to describe the scheduling logic of the OmniPipe scheduling framework. The algorithm can accept pipeline parallelism P, micro-batch size N, the current process's Stage number (rank), and the optimized scheduling policy as input parameters. The core of the algorithm lies in its support for three different computational strategies, enabling dynamic scheduling based on memory status and computational requirements. The algorithm is shown in Table 1 below:

[0106]

[0107] Table 1. A Model Training Strategy Scheduling Algorithm

[0108] Specifically, the forward computation phase includes the implementation of three computation strategies:

[0109] Standard strategy (None). Performs the standard forward computation process, saving all intermediate activation values ​​for subsequent backpropagation. This strategy is suitable when GPU memory resources are sufficient, requiring no additional computational overhead, but it consumes a significant amount of GPU memory.

[0110] Activation value recompute strategy (Recompute). This strategy performs forward computation but does not save intermediate results, implemented through the "forwardwithout save" interface, skipping the activation value storage step. This strategy significantly reduces memory usage, but requires recompiling during backpropagation, increasing computational overhead.

[0111] Activation value transfer strategy. This strategy uses a custom activation hook to capture intermediate results and implements asynchronous transfer of activation values ​​through a "lazy transfer" mechanism. This approach maintains low memory usage while avoiding the overhead of recalculation.

[0112] Correspondingly, the reverse computation phase also supports three strategies:

[0113] The standard strategy (None) directly uses the stored activation values ​​to perform backpropagation, calculate gradients, and update parameters. This strategy has the highest memory overhead but the highest computational efficiency.

[0114] The activation value recompute strategy involves first re-performing the forward computation to restore the required intermediate states, followed immediately by performing the backward propagation computation. This strategy trades increased computation time for increased GPU memory.

[0115] Activation value transfer strategy. The required activation values ​​are pre-fetched using a "pre-fetch" operation, ensuring backpropagation is performed once the data is ready. This strategy balances computational efficiency and memory usage.

[0116] Based on the above, this application provides a method for generating pipelined parallel training scheduling strategies. The model can be divided into multiple stages by layers, with each stage deployed on different computing devices. The model's training data flows through each stage in micro-batches. For example... Figure 3 As shown, the method includes the following steps.

[0117] Step S301: Obtain the configuration parameters for model training. The model training adopts the first scheduling strategy, which is a pipelined parallel 1F1B scheduling strategy.

[0118] Step S302: Calculate the memory usage of the computing device based on the configuration parameters.

[0119] Step S303: If the predicted memory usage exceeds the memory usage threshold, model the pipeline parallelism. This modeling includes establishing micro-batch dependency constraints, memory window constraints, and activation value transfer quantity constraints for pipeline parallelism.

[0120] Step S304: Generate a second scheduling strategy based on the modeling results.

[0121] Model training configuration parameters refer to all adjustable variables that need to be set before the training process begins, controlling how the model learns from the data and ultimately affecting the model's performance.

[0122] In one implementation, the above modeling can be solved using a commercial solver to generate a scheduling strategy.

[0123] In one implementation, the configuration parameters may include model configuration parameters, computing device cluster configuration parameters, and pipeline parallelism configuration parameters.

[0124] In one implementation, the pipeline can be divided into a preheating state, a stable training state, and a cooling state.

[0125] Specifically, during the preheating phase, the micro-batch dependency constraint can be configured such that, within the same stage, the start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of the previous micro-batch. In different stages, the start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of the same micro-batch in the previous stage.

[0126] In a stable training state, the micro-batch dependency constraint can be configured as follows: In different alternating cycles, the start time of the forward propagation of the current micro-batch is later than the end time of the back propagation of the current micro-batch in the previous cycle. Here, an alternating cycle refers to a process of performing one forward propagation and one back propagation in one phase. The start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of that micro-batch in the previous phase. The start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of that micro-batch in the previous cycle. The start time of the back propagation of the current micro-batch is later than the end time of the back propagation of that micro-batch in the next phase. In the same alternating cycle, the start time of the back propagation of the current micro-batch is later than the end time of the forward propagation of the current micro-batch in the current cycle.

[0127] In the cooling state, the micro-batch dependency constraint can be configured such that, in different stages, the start time of backpropagation for the current micro-batch is later than the end time of backpropagation for the same micro-batch in the next stage. The start time of backpropagation for the current micro-batch is later than the end time of backpropagation for the same micro-batch in the previous loop.

[0128] In one implementation, the memory window constraint may include: within each window, the number of cached micro-batches does not exceed the maximum number of micro-batches that can be stored in the current stage.

[0129] In one implementation, the activation value transfer quantity constraint may include: the total amount of activation value transfers within any window shall not exceed the storage capacity of the current activation value sender, nor the storage capacity of the current activation value receiver.

[0130] Based on the above, this application provides a method for applying a pipelined parallel training scheduling strategy. The model can be divided into multiple stages by layers, with each stage deployed on a different computing device. The model's training data flows through each stage in micro-batches. For example... Figure 4 As shown, the method includes the following steps.

[0131] Step S401: Obtain the configuration parameters for model training. The model training adopts the first scheduling strategy, which is a pipelined parallel 1F1B scheduling strategy.

[0132] Step S402: Calculate the memory usage of the computing device based on the configuration parameters.

[0133] Step S403: If the predicted memory usage exceeds a preset memory threshold, a second scheduling strategy is obtained. The second scheduling strategy is obtained by modeling and solving the pipeline parallelism. The modeling includes establishing micro-batch dependency constraints, memory window constraints, and activation value transfer quantity constraints for pipeline parallelism.

[0134] Step S404: The model training is scheduled using the second scheduling strategy. The scheduling objects are micro-batches and / or activation values.

[0135] Step S405: If the predicted memory usage result is less than or equal to the preset memory threshold, continue to execute the first strategy.

[0136] Model training configuration parameters refer to all adjustable variables that need to be set before the training process begins, controlling how the model learns from the data and ultimately affecting the model's performance.

[0137] In one implementation, the configuration parameters may include model configuration parameters, computing device cluster configuration parameters, and pipeline parallelism configuration parameters.

[0138] In one implementation, the pipeline can be divided into a preheating state, a stable training state, and a cooling state.

[0139] Specifically, during the preheating phase, the micro-batch dependency constraint can be configured such that, within the same stage, the start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of the previous micro-batch. In different stages, the start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of the same micro-batch in the previous stage.

[0140] In a stable training state, the micro-batch dependency constraint can be configured as follows: In different alternating cycles, the start time of the forward propagation of the current micro-batch is later than the end time of the back propagation of the current micro-batch in the previous cycle. Here, an alternating cycle refers to a process of performing one forward propagation and one back propagation in one phase. The start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of that micro-batch in the previous phase. The start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of that micro-batch in the previous cycle. The start time of the back propagation of the current micro-batch is later than the end time of the back propagation of that micro-batch in the next phase. In the same alternating cycle, the start time of the back propagation of the current micro-batch is later than the end time of the forward propagation of the current micro-batch in the current cycle.

[0141] In the cooling state, the micro-batch dependency constraint can be configured such that, in different stages, the start time of backpropagation for the current micro-batch is later than the end time of backpropagation for the same micro-batch in the next stage. The start time of backpropagation for the current micro-batch is later than the end time of backpropagation for the same micro-batch in the previous loop.

[0142] In one implementation, the memory window constraint may include: within each window, the number of cached micro-batches does not exceed the maximum number of micro-batches that can be stored in the current stage.

[0143] In one implementation, the activation value transfer quantity constraint may include: the total amount of activation value transfers within any window shall not exceed the storage capacity of the current activation value sender, nor the storage capacity of the current activation value receiver.

[0144] Based on the above description, embodiments of this application also provide a model training scheduling strategy generation apparatus. For example... Figure 5 As shown, the model training scheduling strategy generation device may include:

[0145] The first acquisition module 510 acquires the configuration parameters for model training. The model training adopts the first scheduling strategy, which is a pipelined parallel 1F1B scheduling strategy.

[0146] The first prediction module 520 is used to predict the memory usage of the computing device based on the configuration parameters.

[0147] The first judgment module 530 is used to determine whether the current video memory usage prediction result exceeds the preset video memory threshold.

[0148] Modeling module 540 models pipeline parallelism when the predicted memory usage exceeds the memory usage threshold. This modeling includes establishing micro-batch dependency constraints, memory window constraints, and activation value transfer quantity constraints for pipeline parallelism.

[0149] The scheduling strategy generation module 550 is used to generate a second scheduling strategy based on the modeling results.

[0150] In one implementation, the configuration parameters may include model configuration parameters, computing device cluster configuration parameters, and pipeline parallelism configuration parameters.

[0151] In one implementation, the modeling module 540 can be used to divide the pipeline into a preheating state, a stable training state, and a cooling state.

[0152] Specifically, during the preheating phase, the micro-batch dependency constraint can be configured such that, within the same stage, the start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of the previous micro-batch. In different stages, the start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of the same micro-batch in the previous stage.

[0153] In a stable training state, the micro-batch dependency constraint can be configured as follows: In different alternating cycles, the start time of the forward propagation of the current micro-batch is later than the end time of the back propagation of the current micro-batch in the previous cycle. Here, an alternating cycle refers to a process of performing one forward propagation and one back propagation in one phase. The start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of that micro-batch in the previous phase. The start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of that micro-batch in the previous cycle. The start time of the back propagation of the current micro-batch is later than the end time of the back propagation of that micro-batch in the next phase. In the same alternating cycle, the start time of the back propagation of the current micro-batch is later than the end time of the forward propagation of the current micro-batch in the current cycle.

[0154] In the cooling state, the micro-batch dependency constraint can be configured such that, in different stages, the start time of backpropagation for the current micro-batch is later than the end time of backpropagation for the same micro-batch in the next stage. The start time of backpropagation for the current micro-batch is later than the end time of backpropagation for the same micro-batch in the previous loop. In one implementation,

[0155] In one implementation, the memory window constraint may include: within each window, the number of cached micro-batches does not exceed the maximum number of micro-batches that can be stored in the current stage.

[0156] In one implementation, the activation value transfer quantity constraint may include: the total amount of activation value transfers within any window shall not exceed the storage capacity of the current activation value sender, nor the storage capacity of the current activation value receiver.

[0157] Based on the above description, embodiments of this application also provide a scheduling strategy application device. For example... Figure 6 The strategy application device shown may include:

[0158] The second acquisition module 610 is used to acquire the configuration parameters for model training. The model training adopts the first scheduling strategy, which is a pipelined parallel 1F1B scheduling strategy.

[0159] The second prediction module 620 is used to predict the memory usage of the computing device based on the configuration parameters.

[0160] The second judgment module 630 is used to determine whether the current video memory usage prediction result exceeds the preset video memory threshold.

[0161] The scheduling strategy acquisition module 640 acquires a second scheduling strategy when the predicted memory usage exceeds the memory usage threshold. This second scheduling strategy is based on the results of pipeline parallel modeling. The modeling includes establishing micro-batch dependency constraints, memory window constraints, and activation value transfer quantity constraints for pipeline parallelism.

[0162] The second scheduling module 650 uses a second scheduling strategy to schedule model training. The scheduling objects are micro-batches and / or activation values.

[0163] The first scheduling module 660 continues to execute the first scheduling strategy if the predicted memory usage result is less than or equal to the memory usage threshold.

[0164] In one implementation, the configuration parameters may include model configuration parameters, computing device cluster configuration parameters, and pipeline parallelism configuration parameters.

[0165] In one implementation, the above-mentioned pipeline parallel modeling includes: dividing the pipeline into a preheating state, a stable training state, and a cooling state.

[0166] Specifically, during the preheating phase, the micro-batch dependency constraint can be configured such that, within the same stage, the start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of the previous micro-batch. In different stages, the start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of the same micro-batch in the previous stage.

[0167] In a stable training state, the micro-batch dependency constraint can be configured as follows: In different alternating cycles, the start time of the forward propagation of the current micro-batch is later than the end time of the back propagation of the current micro-batch in the previous cycle. Here, an alternating cycle refers to a process of performing one forward propagation and one back propagation in one phase. The start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of that micro-batch in the previous phase. The start time of the forward propagation of the current micro-batch is later than the end time of the forward propagation of that micro-batch in the previous cycle. The start time of the back propagation of the current micro-batch is later than the end time of the back propagation of that micro-batch in the next phase. In the same alternating cycle, the start time of the back propagation of the current micro-batch is later than the end time of the forward propagation of the current micro-batch in the current cycle.

[0168] In the cooling state, the micro-batch dependency constraint can be configured such that, in different stages, the start time of backpropagation for the current micro-batch is later than the end time of backpropagation for the same micro-batch in the next stage. The start time of backpropagation for the current micro-batch is later than the end time of backpropagation for the same micro-batch in the previous loop. In one implementation,

[0169] In one implementation, the memory window constraint may include: within each window, the number of cached micro-batches does not exceed the maximum number of micro-batches that can be stored in the current stage.

[0170] In one implementation, the activation value transfer quantity constraint may include: the total amount of activation value transfers within any window shall not exceed the storage capacity of the current activation value sender, nor the storage capacity of the current activation value receiver.

[0171] Those skilled in the art will readily recognize that, based on the units and algorithm steps described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0172] Corresponding to the methods provided by the present invention, the present invention also provides a computer-readable storage medium. This computer-readable storage medium includes a stored program, wherein the program, when executed by a processor, implements the methods described in any of the above embodiments.

[0173] This application also provides a computer program product that, when run on a computer, causes the computer to execute the relevant method steps described in the method embodiments above. This computer program product can be stored on a readable storage medium.

[0174] The aforementioned storage media include: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks, and other media that can store program code.

[0175] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0176] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating a model training scheduling strategy, wherein the model is divided into multiple stages by layers, each stage is deployed on a different computing device, and training data flows through each stage in micro-batch form, characterized in that, The method includes: Obtain the configuration parameters for model training, wherein the model training adopts a first scheduling strategy, which is a pipelined parallel 1F1B scheduling strategy; Predict the video memory usage of the computing device based on the configuration parameters; If the predicted memory usage exceeds the memory usage threshold, the pipeline parallelism is modeled; wherein, the modeling includes establishing micro-batch dependency constraints, memory window constraints, and activation value transfer quantity constraints for pipeline parallelism. A second scheduling strategy is generated based on the modeling results.

2. The method according to claim 1, characterized in that, The configuration parameters include model configuration parameters, computing device cluster configuration parameters, and pipeline parallel configuration parameters.

3. The method according to claim 1, characterized in that, The parallel modeling of the pipeline includes: The pipeline is divided into three parallel states: a warm-up state, a stable training state, and a cooling state. In the preheating state, the micro-batch dependency constraint is configured as follows: within the same stage, the forward propagation start time of this micro-batch is later than the forward propagation end time of the previous micro-batch; in different stages, the forward propagation start time of this micro-batch is later than the forward propagation end time of the micro-batch in the previous stage. In the stable training state, the micro-batch dependency constraint is configured as follows: In different alternating cycles, the start time of the forward propagation of the current micro-batch is later than the end time of the backward propagation of the current micro-batch in the previous cycle; where alternating cycles refer to a process of performing one forward propagation and one backward propagation in one phase. The start time of the forward propagation of the micro-batch in this stage is later than the end time of the forward propagation of the same micro-batch in the previous stage; The start time of the forward propagation of the micro-batch in this stage is later than the end time of the forward propagation of the micro-batch in the previous cycle; The start time of the back propagation of the micro-batch in this stage is later than the end time of the back propagation of the micro-batch in the next stage. In the same alternating cycle, the start time of the back propagation of the micro-batch in this stage is later than the end time of the forward propagation of the micro-batch in this stage in this cycle; In the cooled state, the micro-batch dependency constraint is configured such that, in different stages, the start time of the backpropagation of the micro-batch in this stage is later than the end time of the backpropagation of the micro-batch in the next stage; and the start time of the backpropagation of the micro-batch in this stage is later than the end time of the backpropagation of the micro-batch in this stage in the previous loop.

4. The method according to claim 1, characterized in that, The memory window constraint includes: within each window, the number of cached micro-batches does not exceed the maximum number of micro-batches that can be stored in the current stage.

5. The method according to claim 1, characterized in that, The activation value transfer quantity constraint includes: the total amount of activation value transfers within any window shall not exceed the storage capacity of the current activation value sender, and shall not exceed the storage capacity of the current activation value receiver.

6. A method for applying a scheduling strategy, characterized in that, include: Obtain the configuration parameters for model training, wherein the model training adopts a first scheduling strategy, which is a pipelined parallel 1F1B scheduling strategy; Predict the video memory usage of the computing device based on the configuration parameters; If the predicted memory usage exceeds the memory usage threshold, a second scheduling strategy is obtained. The second scheduling strategy is based on the pipeline parallel modeling results. The modeling includes establishing micro-batch dependency constraints, memory window constraints, and activation value transfer quantity constraints for pipeline parallelism. The second scheduling strategy is used to schedule model training; wherein the scheduling object is a micro-batch and / or activation value; If the predicted memory usage is less than or equal to the memory usage threshold, the first scheduling strategy continues to be executed.

7. The method according to claim 6, characterized in that, The configuration parameters include model configuration parameters, computing device cluster configuration parameters, and pipeline parallel configuration parameters.

8. The method according to claim 6, characterized in that, The parallel modeling of the pipeline includes: The pipeline is divided into three parallel states: a warm-up state, a stable training state, and a cooling state. In the preheating state, the micro-batch dependency constraint is configured as follows: within the same stage, the forward propagation start time of this micro-batch is later than the forward propagation end time of the previous micro-batch; in different stages, the forward propagation start time of this micro-batch is later than the forward propagation end time of the micro-batch in the previous stage. In the stable training state, the micro-batch dependency constraint is configured as follows: In different alternating cycles, the start time of the forward propagation of the current micro-batch is later than the end time of the backward propagation of the current micro-batch in the previous cycle; where alternating cycles refer to a process of performing one forward propagation and one backward propagation in one phase. The start time of the forward propagation of the micro-batch in this stage is later than the end time of the forward propagation of the same micro-batch in the previous stage; The start time of the forward propagation of the micro-batch in this stage is later than the end time of the forward propagation of the micro-batch in the previous cycle; The start time of the back propagation of the micro-batch in this stage is later than the end time of the back propagation of the micro-batch in the next stage. In the same alternating cycle, the start time of the back propagation of the micro-batch in this stage is later than the end time of the forward propagation of the micro-batch in this stage in this cycle; In the cooled state, the micro-batch dependency constraint is configured such that, in different stages, the start time of the backpropagation of the micro-batch in this stage is later than the end time of the backpropagation of the micro-batch in the next stage; and the start time of the backpropagation of the micro-batch in this stage is later than the end time of the backpropagation of the micro-batch in this stage in the previous loop.

9. The method according to claim 6, characterized in that, The memory window constraint includes: within each window, the number of cached micro-batches does not exceed the maximum number of micro-batches that can be stored in the current stage.

10. The method according to claim 6, characterized in that, The activation value transfer quantity constraint includes: the total amount of activation value transfers within any window shall not exceed the storage capacity of the current activation value sender, nor the storage capacity of the current activation value receiver.

Images