Large model training method and device based on staggered pipeline parallel strategy and computer equipment

Abstract

The invention relates to a large model training method and device based on a staggered pipeline parallel strategy and computer equipment. The method comprises the steps that on the basis of batch data in a training set, a staggered assembly line parallel strategy is adopted for training of a large model, the staggered assembly line parallel strategy comprises the steps that the assembly line parallel iteration process is divided into a preheating stage, a stabilizing stage and a cooling stage, each calculation unit executes N forward calculation tasks in the preheating stage, and each calculation unit executes N forward calculation tasks in the stabilizing stage; wherein N = streamline parallel scale * block number-phase value corresponding to the calculation unit, the streamline parallel scale and the total number of the calculation unit are M, and the phase value corresponding to the calculation unit is an integer greater than or equal to 0 and less than M; and each calculation unit alternately executes the remaining forward calculation tasks among the reverse calculation tasks in pairs in the stable stage, and executes the remaining reverse calculation tasks in the cooling stage. By adopting the method, the training efficiency of the large model can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a large model training method, apparatus and computer equipment based on an interleaved pipeline parallel strategy. Background Art

[0002] In the field of artificial intelligence, training large models places extremely high demands on computing device performance, particularly on GPU (Graphics Processing Unit) memory. During large model training, device memory requirements are determined by the peak memory usage of a particular computing device during training, which often occurs when activation values accumulate significantly. Activation values are dynamically generated during model training, generated during forward computation and temporarily stored in memory for use in the model's backward propagation. The corresponding memory is released only after the backward propagation is complete.

[0003] To improve the training efficiency of large models, pipeline parallelism strategies are widely adopted in the industry. Mainstream strategies include non-interleaved pipeline parallelism (One Forward Pass followed by One Backward Pass, 1F1B) and interleaved pipeline parallelism (interleave 1F1B). The interleave 1F1B strategy effectively reduces computational bubbles during training by splitting the training process at a finer granularity, significantly improving training performance.

[0004] However, while the interleaved pipeline parallel strategy improves performance, it also significantly increases the peak memory usage of the device with the highest memory load (device 1) during model training. This significantly limits the application of the interleaved 1F1B strategy on devices with limited memory resources, preventing it from fully leveraging its advantages in improving training performance. This has become a technical bottleneck hindering the efficient training of large models. Summary of the Invention

[0005] Based on this, it is necessary to provide a large model training method, device and computer equipment based on an interleaved pipeline parallel strategy that can reduce peak video memory during large model training using an interleaved pipeline parallel strategy to address the above technical problems.

[0006] In a first aspect, the present application provides a large model training method based on an interleaved pipeline parallel strategy, the method comprising:

[0007] Based on the batch data in the training set, a staggered pipeline parallel strategy is used to train the large model, wherein the staggered pipeline parallel strategy includes:

[0008] The pipeline parallel iterative process is divided into a warm-up phase, a stabilization phase, and a cooling phase. Each computing unit performs N forward computing tasks in the warm-up phase, where:

[0009] N = pipeline parallel scale × number of blocks - stage value corresponding to the computing unit, where the pipeline parallel scale and the total number of computing units are both M, and the stage value corresponding to the computing unit is an integer greater than or equal to 0 and less than M;

[0010] Each of the computing units interleaves the remaining forward computing tasks with the reverse computing tasks in pairs during the stabilization phase, and executes the remaining reverse computing tasks during the cooling phase.

[0011] In one embodiment, after completing the forward computation task of a target micro-batch in the last block, the reverse computation task of the target micro-batch in the last block is immediately executed, where the target micro-batch is the first micro-batch in the first micro-batch group.

[0012] In one embodiment, after executing the forward computation task of the target micro-batch in the last block, the reverse computation task of the target micro-batch in the last block is executed after one forward computation task, and the target micro-batch is the first micro-batch in the first micro-batch group.

[0013] In one embodiment, the cooling phase executes N reverse computing tasks, and the number of reverse computing tasks and forward computing tasks executed in the stabilization phase are both the total amount of batch data × the number of blocks - N.

[0014] In one embodiment, the training of a large model using an interleaved pipeline parallel strategy includes:

[0015] Obtaining a batch data group size and a number of blocks, dividing the batch data into a plurality of micro-batch groups based on the group size, each micro-batch group including group size micro-batches, and dividing the stages in the pipeline parallelism based on the number of blocks to obtain a plurality of blocks, each block including pipeline parallelism size virtual stages;

[0016] During the training process of the large model, the micro-batches within each micro-batch group are forward-calculated on each computing unit in pipeline order. The forward calculation between each micro-batch group is executed serially, the forward calculation and reverse calculation between each block are executed serially, and the reverse calculation of the micro-batches within the same block is executed continuously.

[0017] In one embodiment, the ratio of the batch data size to the pipeline parallel scale is a positive integer greater than or equal to 2, and the pipeline parallel scale is greater than 2.

[0018] In a second aspect, the present application further provides a large model training device based on an interleaved pipeline parallel strategy, comprising:

[0019] The training module is used to train the large model based on the batch data in the training set using an interleaved pipeline parallel strategy, wherein the interleaved pipeline parallel strategy includes:

[0020] The pipeline parallel iterative process is divided into a warm-up phase, a stabilization phase, and a cooling phase. Each computing unit performs N forward computing tasks in the warm-up phase, where:

[0021] N = pipeline parallel scale × number of blocks - stage value corresponding to the computing unit, where the pipeline parallel scale and the total number of computing units are both M, and the stage value corresponding to the computing unit is an integer greater than or equal to 0 and less than M;

[0022] Each of the computing units interleaves the remaining forward computing tasks with the reverse computing tasks in pairs during the stabilization phase, and executes the remaining reverse computing tasks during the cooling phase.

[0023] In one embodiment, after completing the forward computation task of a target micro-batch in the last block, the reverse computation task of the target micro-batch in the last block is immediately executed, where the target micro-batch is the first micro-batch in the first micro-batch group.

[0024] In one embodiment, after executing the forward computation task of the target micro-batch in the last block, the reverse computation task of the target micro-batch in the last block is executed after one forward computation task, and the target micro-batch is the first micro-batch in the first micro-batch group.

[0025] In one embodiment, the cooling phase executes N reverse computing tasks, and the number of reverse computing tasks and forward computing tasks executed in the stabilization phase are both the total amount of batch data × the number of blocks - N.

[0026] In one embodiment, the training of a large model using an interleaved pipeline parallel strategy includes:

[0027] Obtaining a batch data group size and a number of blocks, dividing the batch data into a plurality of micro-batch groups based on the group size, each micro-batch group including group size micro-batches, and dividing the stages in the pipeline parallelism based on the number of blocks to obtain a plurality of blocks, each block including pipeline parallelism size virtual stages;

[0028] During the training process of the large model, the micro-batches within each micro-batch group are forward-calculated on each computing unit in pipeline order. The forward calculation between each micro-batch group is executed serially, the forward calculation and reverse calculation between each block are executed serially, and the reverse calculation of the micro-batches within the same block is executed continuously.

[0029] In one embodiment, the ratio of the batch data size to the pipeline parallel scale is a positive integer greater than or equal to 2, and the pipeline parallel scale is greater than 2.

[0030] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements any of the above large model training methods based on the interleaved pipeline parallel strategy.

[0031] In a fourth aspect, the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements any of the above large model training methods based on the interleaved pipeline parallel strategy.

[0032] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above large model training methods based on an interleaved pipeline parallel strategy.

[0033] The above-mentioned large model training method, device, computer equipment, computer-readable storage medium and computer program product based on the staggered pipeline parallel strategy can be used to train large models based on batch data in the training set using the staggered pipeline parallel strategy, wherein the staggered pipeline parallel strategy includes: dividing the pipeline parallel iterative process into a warm-up stage, a stabilization stage and a cooling stage, and each computing unit performs N forward computing tasks in the warm-up stage, wherein: N = pipeline parallel scale × number of blocks - stage value corresponding to the computing unit, the pipeline parallel scale and the total number of computing units are both M, and the stage value corresponding to the computing unit is an integer greater than or equal to 0 and less than M. In the stabilization stage, each computing unit intersperses the remaining forward computing tasks with the reverse computing tasks, and executes them in pairs alternately, and executes the remaining reverse computing tasks in the cooling stage. The large model training method, apparatus, computer equipment, computer-readable storage medium and computer program product based on the staggered pipeline parallel strategy provided by the embodiments of the present application reduce the forward computing tasks performed by the computing device with the highest video memory load in the warm-up phase, and the remaining forward computing tasks in the stable phase are interspersed and executed alternately in pairs between the reverse computing tasks, that is, at most one forward computing task is performed before the first reverse computing in the stable phase, which overall reduces the number of forward computing tasks performed by the peak video memory device before the first reverse computing task, avoids the video memory surge problem caused by the concentration of forward tasks on the highest-loaded device in the traditional staggered pipeline strategy, reduces the peak video memory of the highest-loaded device in the system (such as device 1), reduces the restrictions of the video memory bottleneck on the computing device, and enables the pipeline parallel strategy to run efficiently on devices with tight video memory resources, breaking through the device applicability limitations of the traditional strategy, while maintaining high-performance computing, achieving balanced utilization of computing resources and storage resources, and improving the efficiency of large model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 is a schematic diagram of a traditional staggered assembly line refrigerator strategy in one embodiment;

[0036] Figure 2 1 is a flow chart of a large model training method based on an interleaved pipeline parallel strategy in one embodiment;

[0037] Figure 3A schematic diagram of a large model training process using an interleaved pipeline parallel strategy in one embodiment;

[0038] Figure 4 Schematic diagram of an interleaved pipeline parallel strategy in one embodiment;

[0039] Figure 5 is a schematic diagram of an interleaved pipeline parallel strategy in another embodiment;

[0040] Figure 6 A schematic diagram showing a comparison of staggered pipeline parallel strategies in one embodiment;

[0041] Figure 7 is a comparative schematic diagram of an interleaved pipeline parallel strategy in another embodiment;

[0042] Figure 8 is a comparative schematic diagram of an interleaved pipeline parallel strategy in another embodiment;

[0043] Figure 9 1 is a structural block diagram of a large model training device based on an interleaved pipeline parallel strategy in one embodiment;

[0044] Figure 10 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0046] Interleaved 1F1B is a pipeline parallel strategy for efficiently training ultra-large-scale models in a multi-artificial intelligence chip environment, where the artificial intelligence chip can be any one of a GPU (Graphics Processing Unit), a TPU (Tensor Processing Unit), an NPU (Neural Network Processing Unit), a DPU (Deep Learning Processing Unit), an APU (Accelerated Processing Unit), and a GPGPU (General-Purpose Graphics Processing Unit). This is not specifically limited in the embodiments of the present application, and the following explanation will be made using GPU as an example.

[0047] For very large models, a single GPU cannot accommodate all parameters. Therefore, the model needs to be split into multiple parts and placed on different GPUs. This is known as model parallelism. Pipeline parallelism involves hierarchically dividing the model into multiple stages, each deployed on an independent device. Data is sequentially passed through these stages to complete forward and backward computations. The core principle is staggered execution. After the initial warm-up phase, each stage in the pipeline sequentially performs forward propagation of one micro-batch (the entire training batch is divided into several smaller micro-batches, each of which is run independently in the pipeline), followed by backward propagation of another micro-batch. For example, at a certain stage, after the forward propagation of the first micro-batch completes, the system immediately performs backward propagation on it, while the forward propagation of the second micro-batch begins at the previous stage. This staggered forward and backward computation of each stage reduces computation bubbles in the pipeline. Staggered execution allows computation and communication to overlap. While one micro-batch is performing backward propagation, the previous stage can simultaneously perform forward propagation of the next micro-batch. This allows for partial temporal overlap between data transfer and computation, effectively masking communication latency. By alternating forward and reverse operations, each GPU is kept working most of the time, reducing resource waste caused by pipeline waiting.

[0048] The traditional staggered pipeline parallel strategy includes: dividing the pipeline parallel iteration process into a warm-up phase, a stabilization phase, and a cooling phase. The warm-up phase is a phase in which only forward calculations are performed, the stabilization phase is a phase in which forward calculations and reverse calculations are performed in pairs, and the cooling phase is a phase in which only reverse calculations are performed. The number of forward calculation tasks performed by each computing unit in the warm-up phase is (pipeline parallel scale - stage value corresponding to the computing unit) × 2 + (number of blocks - 1) × batch data grouping size. The meaning of each parameter is combined in the following example. Figure 1 Provide explanation.

[0049] For example, Figure 1 A schematic diagram of a traditional interleaved pipeline parallel strategy is shown. Figure 1The four compute devices (device1 - device4) represent different computational units in the pipeline's parallelism. The number of microbatches (num_microbatches) is 8, representing the total number of microbatches divided during the computation, with batch numbers 1, 2, 3, 4, 5, 6, 7, and 8. The pipeline parallelism scale (also known as the number of pipeline stages, pp_stage) matches the number of compute devices, which is 4, indicating that the pipeline is divided into four stages (large models are divided into four stages per layer). The number of chunks (also known as the chunk size, chunk_size) is 2, meaning that each stage is further divided into two virtual stages, chunk1 and chunk2, for a total of eight virtual stages. (Chunk1 includes stages 0 through 3 (computed sequentially on devices 1 through 4), and chunk2 includes stages 4 through 7 (computed sequentially on devices 1 through 4). The number of groups (num_group) is the ratio of the total number of microbatches to the parallel scale of the pipeline (num_microbatches / pp_stage). In this example, it is 2, which means that the 8 microbatches are divided into 2 groups. Group 1 (including microbatches 1 to 4) Figure 1 1 to 4) and group2 (including 5 to 8 micro-batches, Figure 1 5 to 8 respectively).

[0050] exist Figure 1 In the example shown, the number of forward computation tasks executed by each compute unit during the warm-up phase is: device1 = 10, device2 = 8, device3 = 6, and device4 = 4. During the stable phase, the number of forward and reverse computations performed by each compute unit is the same: the total number of computation tasks executed during the entire pipelined parallel iteration process minus the number of forward computation tasks executed during the warm-up phase. The total number of computation tasks executed during the entire pipelined parallel iteration process is equal to the number of micro-batches × the number of blocks. Therefore, the number of forward and reverse computations performed by each compute unit during the stable phase is: device1 = 6, device2 = 8, device3 = 10, and device4 = 12. The number of reverse computation tasks executed during the cool-down phase is: device1 = 10, device2 = 8, device3 = 6, and device4 = 4.

[0051] Figure 1In the example shown, dark blue represents the forward calculation of the first chunk (chunk1), including the forward calculation of four virtual stages (represented as stage 0, stage 1, stage 2 and stage 3 respectively); light blue represents the forward calculation of the second chunk (chunk2), including the forward calculation of four virtual stages (represented as stage 4, stage 5, stage 6 and stage 7 respectively); light green represents the reverse calculation of the second chunk (chunk2), including the reverse calculation of four virtual stages (represented as stage 4, stage 5, stage 6 and stage 7 respectively); dark green represents the reverse calculation of the first chunk (chunk1), including the reverse calculation of four virtual stages (represented as stage 0, stage 1, stage 2 and stage 3 respectively).

[0052] Since activation values are generated during the forward calculation process for subsequent reverse calculation, that is, these activation values will be released after the reverse calculation, the more forward calculations are performed before the reverse calculation, the more activation values are generated, and the more video memory is occupied. The device that occupies the most video memory during the entire pipeline iteration is the video memory peak device, and the moment when the video memory is the highest is the video memory peak. Figure 1 In the example shown, during the pipelined parallel process, each compute unit performs only forward computations during the warmup phase, which accumulates video memory usage. During the stable phase, forward and backward computations are performed alternately. This means that a forward computation is required before the first backward computation. Therefore, the peak video memory usage of each compute unit before the first backward computation is the number of forward computation tasks performed by each compute unit during the warmup phase plus the activation value generated by one forward computation task. Since device 1 performs the most forward computation tasks during the warmup phase, 10 tasks, i.e., forward computations for micro-batches 1-4 in chunk 1 (stage 0) and chunk 2 (stage 4), and micro-batches 5 and 6 in chunk 1 (stage 0), followed by forward computations for micro-batches 7 in chunk 1 (stage 0), during the stable phase, device 1 is the peak video memory device.

[0053] The existence of peak memory devices will greatly limit the application of the traditional interleaved 1F1B strategy on devices with limited memory resources, and cannot fully leverage its advantages in improving training performance, becoming a technical bottleneck restricting the efficient training of large models.

[0054] In an embodiment of the present application, a large model training method based on an interleaved pipeline parallel strategy is provided. By optimizing the arrangement of forward calculations and reverse calculations performed in each stage of the pipeline parallel strategy, the peak video memory of the device with the highest load can be effectively reduced, thereby improving the efficiency of large model training.

[0055] In one embodiment, Figure 2 As shown, a large model training method based on an interleaved pipeline parallel strategy is provided. This embodiment uses the method applied to a terminal as an example. In this embodiment, the method includes the following steps 202, wherein:

[0056] In step 202, based on the batch data in the training set, a staggered pipeline parallel strategy is used to train the large model, wherein the staggered pipeline parallel strategy includes:

[0057] The pipeline parallel iterative process is divided into a warm-up phase, a stabilization phase, and a cooling phase. Each computing unit performs N forward computing tasks in the warm-up phase, where: N = pipeline parallel scale × number of blocks - stage value corresponding to the computing unit. The pipeline parallel scale and the total number of computing units are both M, and the stage value corresponding to the computing unit is an integer greater than or equal to 0 and less than M. In the stabilization phase, each computing unit intersperses the remaining forward computing tasks with the reverse computing tasks in pairs, and executes the remaining reverse computing tasks in the cooling phase.

[0058] In an embodiment of the present application, the batch data in the training set can be divided into multiple micro-batches, and an interleaved pipeline parallel strategy is used based on the multiple micro-batches to train the large model. During the training process, the pipeline parallel iterative process can be divided into a warm-up phase, a stabilization phase, and a cooling phase, wherein the warm-up phase is a phase in which the computing unit only performs forward computing tasks, the stabilization phase is a phase in which forward computing tasks and reverse computing tasks appear in pairs, and the cooling phase is a phase in which the computing unit only performs reverse computing tasks. The number of forward computing tasks to be performed by each computing unit in the warm-up phase is determined by the following formula (1):

[0059] N= PP_stage×chunk_size –PP_stage_num formula (1).

[0060] Where N is the number of forward computing tasks executed by each computing unit, PP_stage is the pipeline parallel scale, chunk_size is the number of chunks, and PP_stage_num is the stage value corresponding to the computing unit. Assume that the pipeline parallel scale is M, that is, the pipeline is divided into M stages, which are executed on M computing units respectively. The stage value of the xth stage executed on the xth computing unit is x-1.

[0061] For example, suppose that Figure 1The parameters of the traditional pipeline parallel strategy shown are consistent, that is, the pipeline parallel scale is 4 and the number of blocks is 2. Therefore, the number of forward computing tasks executed by device 1 during the warm-up phase is 8, the number of forward computing tasks executed by device 2 during the warm-up phase is 7, the number of forward computing tasks executed by device 3 during the warm-up phase is 6, and the number of forward computing tasks executed by device 4 during the warm-up phase is 5. Compared with the traditional pipeline parallel strategy, it is obvious that the computing device with the highest load, device 1, in the method provided by the embodiment of the present application, executes fewer forward tasks during the warm-up phase, resulting in a lower peak load.

[0062] The staggered pipeline parallel strategy provided in the embodiment of the present application is that each computing unit, in the stable phase, intersperses the remaining forward computing tasks with the reverse computing tasks and executes them alternately in pairs. In this way, the computing unit will perform at most one forward computing task before executing the first reverse computing task in the stable phase. Therefore, combined with the warm-up phase, it can be clearly seen that the staggered pipeline parallel strategy provided in the embodiment of the present application is significantly lower in peak memory for devices with higher loads than the traditional staggered pipeline parallel strategy.

[0063] The above-mentioned large model training method based on the staggered pipeline parallel strategy can be used to train large models based on batch data in the training set using the staggered pipeline parallel strategy, wherein the staggered pipeline parallel strategy includes: dividing the pipeline parallel iterative process into a warm-up stage, a stabilization stage, and a cooling stage, and each computing unit performs N forward computing tasks in the warm-up stage, wherein: N = pipeline parallel scale × number of blocks - stage value corresponding to the computing unit, the pipeline parallel scale and the total number of computing units are both M, and the stage value corresponding to the computing unit is an integer greater than or equal to 0 and less than M. In the stabilization stage, each computing unit intersperses the remaining forward computing tasks with the reverse computing tasks, and executes them alternately in pairs, and executes the remaining reverse computing tasks in the cooling stage. The large model training method based on the staggered pipeline parallel strategy provided in the embodiment of the present application reduces the forward computing tasks executed by the computing device with the highest video memory load in the warm-up stage, and the remaining forward computing tasks in the stable stage are interspersed and executed alternately in pairs between the reverse computing tasks, that is, at most one forward computing task is performed before the first reverse computing in the stable stage, which overall reduces the number of forward computing tasks performed by the peak video memory device before the first reverse computing task, avoids the video memory surge problem caused by the concentration of forward tasks on the highest-loaded device in the traditional staggered pipeline strategy, reduces the peak video memory of the highest-loaded device in the system (such as device1), reduces the restrictions of the video memory bottleneck on the computing device, and enables the pipeline parallel strategy to run efficiently on devices with tight video memory resources, breaking through the device applicability limitations of traditional strategies, while maintaining high-performance computing, achieving balanced utilization of computing resources and storage resources, and improving the efficiency of large model training.

[0064] In one example, referring to Figure 3 As shown, the training of a large model using an interleaved pipeline parallel strategy may include the following steps 302 to 304, wherein:

[0065] Step 302: Obtain the batch data group size and the number of blocks. Divide the batch data into multiple micro-batches based on the group size, each micro-batch group includes the group size micro-batches, and divide the stages in the pipeline parallelization based on the number of blocks to obtain multiple blocks, each block including the pipeline parallelization size virtual stages.

[0066] In step 304, during the training of the large model, the micro-batches within each micro-batch group are forward-calculated on each computing unit in a pipeline order. The forward calculations between each micro-batch group are performed serially, the forward calculations and the reverse calculations between each block are performed serially, and the reverse calculations of the micro-batches within the same block are performed continuously.

[0067] In an embodiment of the present application, a batch grouping size can be obtained, where an exemplary grouping size is K. Based on the grouping size, the batch data is divided into a plurality of micro-batch groups, each of which includes K micro-batches. Taking K as 4 and the batch data including 8 micro-batches as an example, two micro-batch groups can be obtained after division, each of which includes 4 micro-batches. The number of blocks is obtained, and each stage in the pipeline parallelism is further divided based on the number of blocks to obtain a plurality of blocks. For example, if the number of blocks is 2, two blocks are obtained after division, each of which contains virtual stages of the pipeline parallelism size.

[0068] During the training of large models, the parallelism principle is as follows: the micro-batches within each micro-batch group are forward-computed on each computing unit in pipeline order, and the forward computation between each micro-batch group is executed serially. That is, the forward computation of the micro-batches within a micro-batch group is executed on each computing unit in pipeline parallel, and the forward computation of the micro-batches within the next micro-batch group is not started until the forward computation of all micro-batches within the micro-batch group is completed. The forward computation and reverse computation between each block are executed serially, and the reverse computation of the micro-batches within the same block is executed continuously. That is, within a micro-batch group, all micro-batches complete the forward computation of all stages of a block before executing the forward computation of the next block, and all micro-batches complete the reverse computation of all stages of the current block before executing the reverse computation of the previous block.

[0069] For example, take 2 micro-batch groups, each of which includes 4 micro-batches (group1 (micro-batches 1-4), group2 (micro-batches 5-8), 2 chunks, and each chunk includes 4 stages (chunk1 (stages 0-3), chunk2 (stages 4-7)). Micro-batches (1-4) in group1 are executed on device1-device4 in pipeline order. After the forward calculation of stages 0-3 of chunk1 is completed, the forward calculation of stages 4-7 of chunk2 is executed. After the forward calculation of micro-batches (1-4) in group1 is completed in chunk1 and chunk2, the forward calculation of micro-batches (5-8) in group2 in chunk1 and chunk2 is executed.

[0070] After the micro-batches in group 1 enter the back propagation phase, the reverse calculation of chunk 2 (7-4) is performed first for micro-batches 1-4 in group 1. After the reverse calculation of chunk 2 (7-4) is completed for all micro-batches (1-4), the reverse calculation of chunk 1 (3-0) is performed.

[0071] In an exemplary embodiment, after the forward computation task of the target micro-batch in the last block is completed, the reverse computation task of the target micro-batch in the last block is immediately executed, and the target micro-batch is the first micro-batch in the first micro-batch group.

[0072] In an embodiment of the present application, after the last computing unit completes the forward computing task of the first micro-batch in the first micro-batch group in the last block, it can immediately execute the reverse computing task of the first micro-batch in the last block, and after completing the reverse computing task of the first micro-batch, the remaining forward computing can be interspersed in the reverse computing tasks in the pipeline order.

[0073] Let’s take the example of 2 micro-batch groups and 2 blocks as an example. Figure 4 As shown, micro-batches (1-4) in group 1 are sequentially executed in pipeline order on devices 1 through 4, performing forward computations on chunk 1's stages 0-3 first, followed by stages 4-7 in chunk 2. After completing the forward computation for micro-batch 1 at stage 7 in chunk 2, reverse computation for that micro-batch 1 is immediately performed, specifically reverse computation for stages 7-4 in chunk 2. These four reverse computations (7-4) are performed continuously. After completing the reverse computation for micro-batch 1 in chunk 2, each computing unit interleaves the remaining forward computation tasks with the reverse computation tasks in pipeline order.

[0074] After the reverse computation of micro-batch 1 in stage 7 is completed on device 4, the remaining forward computation tasks include the forward computation of micro-batches (2-4) in stage 7 and the forward computation of micro-batches (5-8) in stages 3 and 7. The remaining forward computation tasks will be executed alternately with the reverse computation tasks on device 4 in pairs. The computation tasks executed by device 4 in the stable phase are:

[0075] 1(7)bwd->2(7)fwd->2(7)bwd->3(7)fwd->3(7)bwd->4(7)fwd->4(7)bwd->5(3)fwd->1(3)bwd->6(3)fwd->2(3)bwd- >7(3)fwd->3(3)bwd->8(3)fwd->4(3)bwd->5(7)fwd->5(3)bwd->6(7)fwd->6(3)bwd->7(7)fwd->7(3)bwd->8(7)fwd. Among them, the numbers outside the brackets represent the micro-batch number, the numbers inside the brackets represent the stage value, bwd represents the reverse calculation, and fwd represents the forward calculation. For example: 1(7)bwd represents the reverse calculation of micro-batch 1 at stage 7, and 2(7)fwd represents the forward calculation of micro-batch 2 at stage 7.

[0076] Similarly, after the reverse computing task of batch 1 is transferred to phase 2, the computing task performed by device 3 in the stable phase is:

[0077] 1(6)bwd->3(6)fwd->2(6)bwd->4(6)fwd->3(6)bwd->5(2)fwd->4(6)bwd->6(2)fwd->1(2)bwd->7(6)fwd- >2(2)bwd->8(6)fwd->3(2)bwd->5(6)fwd->4(2)bwd->6(6)fwd->5(2)bwd->7(6)fwd->6(2)bwd->8(6)fwd.

[0078] After the reverse computing task of batch 1 is transferred to phase 1, the computing task performed by device 2 in the stable phase is:

[0079] 1(5)bwd->4(5)fwd->2(5)bwd->5(1)fwd->3(5)bwd->6(1)fwd->4(5)bwd->7(1)fwd->1(1)bwd- >8(1)fwd->2(1)bwd->5(5)fwd->3(1)bwd->6(5)fwd->4(1)bwd->7(5)fwd->5(5)bwd->8(1)fwd.

[0080] After the reverse computing task of batch 1 is transferred to phase 0, the computing tasks performed by device 1 in the stable phase are:

[0081] 1(4)bwd->5(0)fwd->2(4)bwd->6(0)fwd->3(4)bwd->7(0)fwd->4(4)bwd->8(0)fwd- >1(0)bwd->5(4)fwd->2(0)bwd->6(4)fwd->3(0)bwd->7(4)fwd->4(0)bwd->8(4)fwd.

[0082] In an exemplary embodiment, after executing the forward computation task of the target micro-batch in the last block, the reverse computation task of the target micro-batch in the last block is executed after one forward computation task. The target micro-batch is the first micro-batch in the first micro-batch group.

[0083] In an embodiment of the present application, after the last computing unit completes the forward computing task of the first micro-batch in the first micro-batch group within the last block, it may execute the forward computing task of the second micro-batch in the first micro-batch group, and then execute the reverse computing task of the first micro-batch in the last block. After completing the reverse computing task of the first micro-batch, the remaining forward computing is interspersed with the reverse computing task in a pipeline order. Each subsequent computing unit executes a forward computing task in a pipeline order before executing the reverse computing task of the first micro-batch in the first micro-batch group, and after completing the reverse computing task of the first micro-batch in the first micro-batch group, the remaining forward computing is interspersed with the reverse computing task in a pipeline order.

[0084] Let’s take the example of 2 micro-batch groups and 2 blocks as an example. Figure 5As shown, micro-batches (1-4) in group 1 are forward-computed on devices 1 through 4 in pipeline order, first executing the forward computation for stages 0-3 in chunk 1, followed by the forward computation for stages 4-7 in chunk 2. After completing the forward computation for stage 7 of micro-batch 1 in chunk 2, the forward computation for micro-batch 2 in chunk 2 is executed. Immediately after completing this forward computation, the reverse computation for micro-batch 1 is performed, i.e., the reverse computation for stages 7-4 of micro-batch 1 in chunk 2. These four reverse computations (7-4) are performed continuously. Each computing device can complete the forward computation for one micro-batch before executing the reverse computation for micro-batch 1 in chunk 2. The remaining forward computation tasks are then interspersed with the remaining reverse computation tasks in pipeline order.

[0085] In one example, N reverse calculation tasks are executed during the cooling phase, and the number of reverse calculation tasks and forward calculation tasks executed during the stabilization phase are both the total amount of batch data × the number of blocks - N. Figure 5 As shown, the forward computing tasks and reverse computing tasks executed by each computing unit in the stable phase are performed alternately in pairs, that is, the number of forward computing tasks and reverse computing tasks executed is the same, then the number of forward computing tasks remaining for each computing unit is the number of forward computing tasks and the number of reverse computing tasks executed in the stable phase, and the remaining number of forward computing tasks is the total amount of batch data × the number of blocks - the number of forward computing tasks executed in the warm-up phase.

[0086] After the forward computation of micro-batch 1 in phase 7 is completed on device 4, the remaining forward computation tasks include the forward computation of micro-batches (2-4) in phase 7 and the forward computation of micro-batches (5-8) in phases 3 and 7. The remaining forward computation tasks will be executed alternately with the backward computation tasks on device 4 in pairs. The computation tasks executed by device 4 in the stable phase are:

[0087] 2(7)fwd->1(7)bwd->3(7)fwd->2(7)bwd->4(7)fwd->3(7)bwd->5(3)fwd->4(7)bwd->6(3)fwd->1(7)bwd->7(3)fwd- >2(7)bwd->8(3)fwd->3(7)bwd->5(7)fwd->4(7)bwd->6(7)fwd->5(3)bwd->7(7)fwd->6(3)bwd->8(7)fwd->7(3)bwd.

[0088] Similarly, before the reverse computation task of batch 1 is transferred to phase 2, device 3 performs a forward computation task. The computation tasks performed during the stable phase are:

[0089] 3(6)fwd->1(6)bwd->4(6)fwd->2(6)bwd->5(2)fwd->3(6)bwd->6(2)fwd->4(6)bwd->7(6)fwd->1(2)bwd- >8(6)fwd->2(2)bwd->5(6)fwd->3(2)bwd->6(6)fwd->4(2)bwd->7(6)fwd->5(2)bwd->8(6)fwd->6(2)bwd.

[0090] Before the reverse computation task of batch 1 is transferred to phase 1, device 2 performs a forward computation task. The computation tasks performed during the stable phase are:

[0091] 4(5)fwd->1(5)bwd->5(1)fwd->2(5)bwd->6(1)fwd->3(5)bwd->7(1)fwd->4(5)bwd->8(1)fwd- >1(1)bwd->5(5)fwd->2(1)bwd->6(5)fwd->3(1)bwd->7(5)fwd->4(1)bwd->8(1)fwd->5(5)bwd.

[0092] Before the reverse computation task of batch 1 is transferred to phase 0, device 1 performs a forward computation task. The computation tasks performed during the stable phase are:

[0093] 5(0)fwd->1(4)bwd->6(0)fwd->2(4)bwd->7(0)fwd->3(4)bwd->8(0)fwd->4(4)bwd- >5(4)fwd->1(0)bwd->6(4)fwd->2(0)bwd->7(4)fwd->3(0)bwd->8(4)fwd->4(0)bwd.

[0094] In this way, the forward calculation task performed by each computing unit before the first reverse calculation in the stable phase is the same as the traditional pipeline parallel strategy, but the number of forward calculation tasks has been reduced in the warm-up phase compared with the traditional pipeline parallel strategy. Therefore, the large model training method based on the staggered pipeline parallel strategy provided in the embodiment of the present application can effectively reduce the peak video memory occupancy of the device with the highest load and improve the efficiency of large model training.

[0095] In an exemplary embodiment, the ratio of the batch data size to the pipeline parallel scale is a positive integer greater than or equal to 2, and the pipeline parallel scale is greater than 2.

[0096] In the traditional pipeline parallel scale strategy, the number of forward computing tasks before the highest-load device performs the first reverse operation is (pp_stage–1)×2+pp_stage×(chunk_size-1)+1. In the embodiment of the present application, the number of forward computing tasks before the highest-load device performs the first reverse operation is pp_stage×chunk_size–pp_stage_num or pp_stage×(chunk_size-1)+2+pp_stage-1. Analysis and statistics show that when the ratio of batch data size to pipeline parallel scale is a positive integer greater than or equal to 2, and the pipeline parallel scale is greater than 2, the peak memory usage of the optimized pipeline parallel scale strategy provided by the embodiment of the present application is lower than that of the traditional pipeline parallel strategy.

[0097] In order to highlight the effect of the embodiment of the present application, respectively Figure 6 、 Figure 7 and Figure 8 The following diagrams respectively show the comparison between the pipeline parallel strategy provided by the embodiment of the present application and the traditional pipeline parallel strategy under different parameters. Figure 6 The parameters are: total number of micro-batch data is 8, pipeline parallel scale is 4, number of blocks is 2, and number of computing units is 4; Figure 7 The parameters are: total number of micro-batch data is 8, pipeline parallel scale is 4, number of blocks is 3, and number of computing units is 4; Figure 8 The parameters are: total number of micro-batch data is 10, pipeline parallel scale is 5, number of blocks is 2, and number of computing units is 5.

[0098] Reference Figure 6 、 Figure 7 and Figure 8 As can be seen from the schematic diagram shown, the embodiment of the present application can reduce the peak video memory of the device with the highest load without increasing the computing bubbles, and can thus run efficiently on devices with tight video memory resources, breaking through the device applicability limitations of traditional strategies, while maintaining high-performance computing, achieving balanced utilization of computing resources and storage resources, and improving the efficiency of large model training.

[0099] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0100] Based on the same inventive concept, the embodiment of the present application also provides a large model training device based on an interleaved pipeline parallel strategy for implementing the large model training method based on an interleaved pipeline parallel strategy involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method. Therefore, the specific limitations of one or more embodiments of the large model training device based on an interleaved pipeline parallel strategy provided below can be found in the above limitations on the large model training method based on an interleaved pipeline parallel strategy, and will not be repeated here.

[0101] In an exemplary embodiment, Figure 9 As shown, a large model training device 900 based on an interleaved pipeline parallel strategy is provided, including: a training module 902, wherein:

[0102] The training module 902 is used to train the large model based on the batch data in the training set using an interleaved pipeline parallel strategy, wherein the interleaved pipeline parallel strategy includes:

[0103] The pipeline parallel iterative process is divided into a warm-up phase, a stabilization phase, and a cooling phase. Each computing unit performs N forward computing tasks in the warm-up phase, where:

[0104] N = pipeline parallel scale × number of blocks - stage value corresponding to the computing unit, where the pipeline parallel scale and the total number of computing units are both M, and the stage value corresponding to the computing unit is an integer greater than or equal to 0 and less than M;

[0105] Each of the computing units interleaves the remaining forward computing tasks with the reverse computing tasks in pairs during the stabilization phase, and executes the remaining reverse computing tasks during the cooling phase.

[0106] The large model training device based on the staggered pipeline parallel strategy provided in the embodiment of the present application reduces the forward computing tasks executed by the computing device with the highest video memory load in the warm-up stage, and the remaining forward computing tasks in the stable stage are interspersed and executed alternately in pairs between the reverse computing tasks, that is, at most one forward computing task is performed before the first reverse computing in the stable stage, which overall reduces the number of forward computing tasks performed by the peak video memory device before the first reverse computing task, avoids the video memory surge problem caused by the concentration of forward tasks on the highest-loaded device in the traditional staggered pipeline strategy, reduces the peak video memory of the highest-loaded device in the system (such as device1), reduces the restrictions of the video memory bottleneck on the computing device, and enables the pipeline parallel strategy to run efficiently on devices with tight video memory resources, breaking through the device applicability limitations of traditional strategies, while maintaining high-performance computing, achieving balanced utilization of computing resources and storage resources, and improving the efficiency of large model training.

[0107] In one embodiment, after completing the forward computation task of a target micro-batch in the last block, the reverse computation task of the target micro-batch in the last block is immediately executed, where the target micro-batch is the first micro-batch in the first micro-batch group.

[0108] In one embodiment, after executing the forward computation task of the target micro-batch in the last block, the reverse computation task of the target micro-batch in the last block is executed after one forward computation task, and the target micro-batch is the first micro-batch in the first micro-batch group.

[0109] In one embodiment, the cooling phase executes N reverse computing tasks, and the number of reverse computing tasks and forward computing tasks executed in the stabilization phase are both the total amount of batch data × the number of blocks - N.

[0110] In one embodiment, the training of a large model using an interleaved pipeline parallel strategy includes:

[0111] Obtaining a batch data group size and a number of blocks, dividing the batch data into a plurality of micro-batch groups based on the group size, each micro-batch group including group size micro-batches, and dividing the stages in the pipeline parallelism based on the number of blocks to obtain a plurality of blocks, each block including pipeline parallelism size virtual stages;

[0112] During the training process of the large model, the micro-batches within each micro-batch group are forward-calculated on each computing unit in pipeline order. The forward calculation between each micro-batch group is executed serially, the forward calculation and reverse calculation between each block are executed serially, and the reverse calculation of the micro-batches within the same block is executed continuously.

[0113] In one embodiment, the ratio of the batch data size to the pipeline parallel scale is a positive integer greater than or equal to 2, and the pipeline parallel scale is greater than 2.

[0114] Each module in the large-scale model training device based on the staggered pipeline parallel strategy can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0115] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 10 As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, and the wireless means can be implemented via Wi-Fi, a mobile cellular network, near-field communication (NFC), or other technologies. When executed by the processor, the computer program implements a large model training method based on an interleaved pipeline parallel strategy. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0116] Those skilled in the art will understand that Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0117] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0118] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0119] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0120] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0121] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0122] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0123] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A large model training method based on an interleaved pipeline parallel strategy, characterized in that: The method comprises: Based on the batch data in the training set, a staggered pipeline parallel strategy is used to train the large model, wherein the staggered pipeline parallel strategy includes: The pipeline parallel iterative process is divided into a warm-up phase, a stabilization phase, and a cooling phase. Each computing unit performs N forward computing tasks in the warm-up phase, where: N = pipeline parallel scale × number of blocks - stage value corresponding to the computing unit, where the pipeline parallel scale and the total number of computing units are both M, and the stage value corresponding to the computing unit is an integer greater than or equal to 0 and less than M; Each of the computing units interleaves the remaining forward computing tasks with the reverse computing tasks in pairs during the stabilization phase, and executes the remaining reverse computing tasks during the cooling phase.

2. The method according to claim 1, characterized in that After completing the forward computation task of the target micro-batch in the last block, immediately executing the reverse computation task of the target micro-batch in the last block, the target micro-batch being the first micro-batch in the first micro-batch group.

3. The method according to claim 1, characterized in that After executing the forward computation task of the target micro-batch in the last block, the reverse computation task of the target micro-batch in the last block is executed after one forward computation task, where the target micro-batch is the first micro-batch in the first micro-batch group.

4. The method according to any one of claims 1 to 3, characterized in that The cooling phase executes N reverse computing tasks, and the number of reverse computing tasks and forward computing tasks executed in the stabilization phase are both the total amount of batch data×the number of blocks-N.

5. The method according to any one of claims 1 to 3, characterized in that The staggered pipeline parallel strategy is used to train large models, including: Obtaining a batch data group size and a number of blocks, dividing the batch data into a plurality of micro-batch groups based on the group size, each micro-batch group including group size micro-batches, and dividing the stages in the pipeline parallelism based on the number of blocks to obtain a plurality of blocks, each block including pipeline parallelism size virtual stages; During the training process of the large model, the micro-batches within each micro-batch group are forward-calculated on each computing unit in pipeline order. The forward calculation between each micro-batch group is executed serially, the forward calculation and reverse calculation between each block are executed serially, and the reverse calculation of the micro-batches within the same block is executed continuously.

6. The method according to any one of claims 1 to 3, characterized in that The ratio of the batch data size to the pipeline parallel scale is a positive integer greater than or equal to 2, and the pipeline parallel scale is greater than 2.

7. A large model training device based on an interleaved pipeline parallel strategy, characterized in that: The device comprises: The training module is used to train the large model based on the batch data in the training set using an interleaved pipeline parallel strategy, wherein the interleaved pipeline parallel strategy includes: The pipeline parallel iterative process is divided into a warm-up phase, a stabilization phase, and a cooling phase. Each computing unit performs N forward computing tasks in the warm-up phase, where: N = pipeline parallel scale × number of blocks - stage value corresponding to the computing unit, where the pipeline parallel scale and the total number of computing units are both M, and the stage value corresponding to the computing unit is an integer greater than or equal to 0 and less than M; Each of the computing units interleaves the remaining forward computing tasks with the reverse computing tasks in pairs during the stabilization phase, and executes the remaining reverse computing tasks during the cooling phase.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images