Bidirectional staggered pipeline parallel method, system and equipment for accelerating large model training

Abstract

The invention discloses a two-way staggered assembly line parallel method for accelerating large model training, and compared with an existing method, the method introduces V-shaped staggered scheduling into a two-way assembly line to form a two-way staggered assembly line parallel mode, and improves the training efficiency of a large model and the use balance of a video memory. A V-shaped staggered assembly line and an inverted V-shaped staggered assembly line are fused to form a bidirectional staggered assembly line, the number of parallel devices is increased, meanwhile, the calculation time of a single micro-batch is shortened, the cavitation rate of the assembly line can be effectively reduced, and more balanced video memory occupation is achieved; in addition, a communication optimization strategy is introduced for overlapping or reducing part of communication overhead, and global gradient synchronization is performed by optimizing pipeline equipment mapping and utilizing a high-speed network in a server, so that the parallel training efficiency of the large model is further improved. The invention further provides a two-way interleaving assembly line parallel system and equipment which are used for achieving the two-way interleaving assembly line parallel method.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a bidirectional staggered pipeline parallel method, system and device for accelerating large model training. Background Art

[0002] In recent years, the field of deep learning has achieved remarkable results by significantly expanding the size of parameters in models. However, training these large models is a very challenging task. The key lies in overcoming the inherent limitations of a single computing device (e.g., GPU) in terms of video memory capacity and computing power, which often become bottlenecks for completing training within a reasonable time limit. Taking the GPT-3 model as an example, the 175 billion parameters require more than 3000GiB of storage space to accommodate the model parameters and optimizer states. If it relies only on a single NVIDIA V100 GPU for training, it will take up to 288 years.

[0003] Faced with this challenge, the importance of distributed parallel training technology has become increasingly prominent. Although data parallelism can provide significant acceleration effects under ideal conditions, its limitations are fully exposed when faced with large models that far exceed the carrying capacity of a single device. Model parallelism came into being. By distributing the huge weight parameters of the model to multiple computing nodes, the video memory burden of a single device is effectively reduced, but the resulting inefficient resource utilization cannot be ignored. In order to further optimize resource utilization, pipeline parallelism divides data batches into micro-batches and divides the model into multiple stages, so that these stages can be executed in parallel on multiple devices, thereby significantly improving training efficiency. For example, the invention application with publication number CN118821972A discloses a model splitting method, device, communication equipment and storage medium. By splitting the initial model into multiple sub-models and pipeline paralleling them on multiple computing acceleration cards, the time-consuming, labor-intensive and error-prone problems of manual model splitting are avoided to improve computing efficiency and resource utilization.

[0004] According to the different weight update mechanisms, pipeline parallelism can be further divided into two categories: synchronous and asynchronous. The synchronous method adheres to strict optimizer semantics and ensures consistency through periodic refreshes, but inevitably introduces device idle periods (i.e., pipeline cavitation), affecting overall efficiency; while the asynchronous method completely eliminates the need for refreshes and reduces cavitation time at the expense of a certain degree of model convergence accuracy. Recently, synchronous pipeline parallelism has made great progress, and strives to maximize training performance by exploring innovative paths such as bidirectional pipeline parallelism and staggered pipeline parallelism. Bidirectional pipeline parallelism effectively balances the consumption of activated video memory and reduces cavitation by allowing each device to carry pipeline stages in multiple directions. The invention application with the publication number CN116755876A discloses a large model hybrid parallel training acceleration method and system, which is based on a more efficient bidirectional pipeline, and balances pipeline utilization and equipment utilization by adjusting the number of micro-batches injected into the pipeline for the first time and the forward calculation of some micro-batches in advance, and uses communication optimization strategies to reduce the communication overhead of hybrid parallel training, thereby improving the overall training efficiency. The staggered pipeline parallelism finely divides the model so that each device processes discontinuous pipeline segments. As the micro-batch processing time is shortened, the bubble size is also reduced.

[0005] Although these efforts have achieved remarkable results, the challenges remain severe: on the one hand, residual cavitation is still a key factor restricting performance improvement, and up to 50% of the time in existing methods may be consumed in pipeline refreshes; on the other hand, even with an efficient point-to-point communication mechanism, pipeline parallelism still faces considerable communication overhead, especially bidirectional pipelines require additional weight storage and data parallel communication overhead, while interleaved pipelines come at the expense of adding more point-to-point communications. Summary of the invention

[0006] The purpose of the present invention is to provide a bidirectional staggered pipeline parallel method for accelerating large model training, which integrates staggered scheduling with bidirectional pipeline to form a bidirectional staggered pipeline parallel mode, effectively reducing the pipeline void rate, achieving a more balanced video memory occupancy, and introducing a communication optimization strategy for overlapping or reducing part of the communication overhead, further improving the efficiency of large model parallel training.

[0007] To achieve the above-mentioned purpose of the invention, the embodiment provides a bidirectional interleaved pipeline parallel method for accelerating large model training, comprising the following steps:

[0008] Using distributed clusters, the large model is sequentially divided into model slices in an interleaved manner, and the model slices are distributed to each device in a V-shaped manner to form a pipeline stage. Each pipeline stage is scheduled sequentially to form a V-shaped interleaved pipeline, and the devices are reversed to form an inverted V-shaped interleaved pipeline.

[0009] By integrating the V-shaped interleaved pipeline and the inverted V-shaped interleaved pipeline, and adjusting the mapping, the devices responsible for the same pipeline stage are placed in the same server, and the two pipelines are scheduled to be located in the same stage of different devices at the same time, forming a bidirectional interleaved pipeline parallel mode;

[0010] An early gradient synchronization strategy is introduced to train the large model in a bidirectional staggered pipeline parallel mode. After the early synchronization conditions are met or the forward and backward calculations of all micro-batches in each pipeline are completed, global gradient synchronization is performed between devices storing the same pipeline stage, and then the parameters of the large model are updated to accelerate the training of the large model.

[0011] In one embodiment, the step of sequentially dividing the large model into model slices in an interlaced manner includes:

[0012] According to the number of pipeline devices P, the large model to be trained is divided into 2P model slices, among which slices 1 to P are divided into the first group, becoming sub-model M1; slices P+1 to 2P are divided into the second group, becoming sub-model M2.

[0013] In one embodiment, the V-shaped staggered pipeline comprises:

[0014] Distribute model slices to each device in a V-shaped manner to form pipeline stages s1 to s 2P , where stages s1 to s P Starting from device G1, the allocation is carried out in stages until device G P , then stage s P+1 to 2P Reverse the order from device G P The pipeline is allocated stage by stage until it returns to device G1, forming a V shape. The pipeline stages are scheduled sequentially to form a V-shaped staggered pipeline.

[0015] In one embodiment, the device is reversely formed into an inverted V-shaped staggered pipeline, comprising:

[0016] Reverse the order of the devices and distribute the model slices to each device in a V-shaped manner to form the pipeline stages s1 to s 2P , where stages s1 to s P From device G P Start allocating in stages until device G1, then stage s P+1 to 2P Reverse the order and allocate from device G1 stage by stage until device G P , forming an inverted V-shaped staggered assembly line.

[0017] In one embodiment, the simultaneous scheduling of two pipelines located at the same stage of different devices includes:

[0018] Different micro-batches or the output results of the previous stage are input at the same time, and the V-shaped and inverted V-shaped staggered pipelines are scheduled in parallel at the same pipeline stage of different devices, forming a bidirectional staggered pipeline parallel mode.

[0019] In one embodiment, the introduction of the advance gradient synchronization strategy includes: optimizing the introduction of the advance synchronization strategy for global gradient synchronization, and advancing the gradient synchronization operation of the sub-model M2 to between the reverse calculations of the last two micro-batches of the sub-model M1 on the device.

[0020] In one embodiment, the early synchronization condition is satisfied when there is a pipeline bubble between the reverse calculations of the last two micro-batches of the sub-model M1 on the device.

[0021] In order to clearly demonstrate the bidirectional interleaved pipeline parallel method for accelerating large model training, the present invention also provides a bidirectional interleaved pipeline parallel system for accelerating large model training, wherein the system uses the bidirectional interleaved pipeline parallel method in the training process, and the bidirectional interleaved pipeline parallel system comprises: a large model segmentation module, a V-shaped interleaved scheduling module, an adjustment device mapping module, a bidirectional interleaved pipeline parallel module, a communication optimization module and a training module;

[0022] The large model segmentation module is used to segment the large model by layer according to the normal training order in an interleaved manner, and divide it into a plurality of stages twice the number of training devices, so that these stages can be executed in parallel on multiple devices;

[0023] The V-shaped staggered scheduling module is used to distribute the model slices to each device in a V-shaped manner to form a pipeline stage;

[0024] The device mapping adjustment module is used to place devices responsible for the same pipeline stage in the same server to optimize the communication bandwidth between the stages;

[0025] The bidirectional interleaved pipeline parallel module is used to parallelly schedule V-shaped and inverted V-shaped interleaved pipelines located at the same stage of different devices, forming a bidirectional interleaved pipeline parallel mode, allowing each device to carry and process pipeline stages in multiple directions;

[0026] The communication optimization module is used to introduce a communication optimization strategy, use the high-speed network in the server to perform global gradient synchronization, and introduce an advance synchronization strategy for global gradient synchronization to optimize it, so that each device can complete the final gradient synchronization closer to the same time;

[0027] The training module is used to train the large model according to the optimized bidirectional interleaved pipeline parallel mode and update the parameters of the large model.

[0028] The present invention also provides a bidirectional interleaved pipeline parallel device for accelerating large model training, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to implement the bidirectional interleaved pipeline parallel method when executing the computer program.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] Compared with the existing methods, the bidirectional staggered pipeline parallel method for accelerating large model training provided by the present invention improves the training efficiency of large models and the balance of video memory usage by introducing V-shaped staggered scheduling into the bidirectional pipeline to form a bidirectional staggered pipeline parallel mode; the bidirectional staggered pipeline is formed by integrating the V-shaped staggered pipeline and the inverted V-shaped staggered pipeline, which increases the number of parallel devices while reducing the calculation time of a single micro-batch, can effectively reduce the pipeline cavitation rate and achieve more balanced video memory occupancy; in addition, a communication optimization strategy is introduced to overlap or reduce part of the communication overhead, and the high-speed network in the server is used to perform global gradient synchronization by optimizing the pipeline device mapping, thereby further improving the parallel training efficiency of large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 It is a flow chart of a bidirectional interleaved pipeline parallel method;

[0033] Figure 2 It is a schematic diagram of the structure of a bidirectional staggered pipeline parallel system;

[0034] Figure 3 It is a schematic diagram of V-type staggered scheduling;

[0035] Figure 4 It is a schematic diagram of a bidirectional interleaved pipeline parallel mode;

[0036] Figure 5 To adjust the pipeline equipment mapping method diagram;

[0037] Figure 6 Schematic diagram of advance gradient synchronization. DETAILED DESCRIPTION

[0038] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0039] To accelerate the training of large models, the embodiment provides a bidirectional interleaved pipeline parallel method for accelerating the training of large models. The method integrates interleaved scheduling with bidirectional pipeline to form a bidirectional interleaved pipeline parallel mode, effectively reducing the pipeline vacancy rate, achieving a more balanced video memory occupancy, and introducing a communication optimization strategy for overlapping or reducing part of the communication overhead, further improving the parallel training efficiency of large models. Figure 1 and 2 As shown, the bidirectional interleaved pipeline parallel method and system provided by the embodiment, wherein the bidirectional interleaved pipeline parallel method comprises the following steps:

[0040] S1. Use distributed clusters to sequentially divide large models into model slices in an interleaved manner, distribute model slices to each device in a V-shaped manner to form a pipeline stage, schedule each pipeline stage in sequence to form a V-shaped interleaved pipeline, and reverse the order of the devices to form an inverted V-shaped interleaved pipeline.

[0041] In the embodiment, the large model is divided into layers according to the normal training order in an interleaved manner. Assuming there are P pipeline devices, it is divided into 2P model slices, of which the first half (slices 1 to P) in the normal training order are divided into the first group, called sub-model M1, and the second half (slices P+1 to 2P) are divided into the second group, called sub-model M2; wherein, the interleaved method refers to dividing the large model into two sub-models, and then dividing each sub-model into slices equal to the number of training devices by layer. Figure 3 As shown in the figure, taking two device pipelines as an example, the large model is divided into four slices, slices 1 and 2 belong to sub-model M1, and slices 3 and 4 belong to sub-model M2. For different devices, the slices communicate through point-to-point; for the same device, the slices communicate through local copies.

[0042] Distribute model slices to each device in a V-shaped manner to form pipeline stages s1 to s 2P , where stages s1 to s P Start with the first device G1 and allocate it in stages until the last device G P , then stage s P+1 ~s 2P Flip sequence device from G P The pipeline is allocated stage by stage until it returns to device G1, forming a V-shaped pipeline. The pipeline stages are scheduled sequentially to form a V-shaped staggered pipeline. Figure 3As shown, the four slices of the large model are sequentially assigned to two devices to form four pipeline stages s1 to s4. Stage s1 is assigned to device G1, stage s2 is assigned to device G2, and then stage s3 is assigned to device G2 in reverse order, and stage s4 is assigned to device G1. Figure 3 As shown in , during training, the calculation of each micro-batch is carried out in stage order, starting from device G1 to device G2, and then continuing from device G2 back to device G1 for forward or reverse calculation. Figure 4 As shown in the figure, taking the four-device pipeline as an example, stages s1 to s4 of the V-type pipeline are respectively assigned to devices G1 to G4, and stages s5 to s8 are respectively assigned to devices G4 to G1. During training, the calculation of each micro-batch starts from device G1 and proceeds to device G4, and then continues from device G4 back to device G1 for forward or reverse calculation.

[0043] Create an inverted V-shaped staggered pipeline in reverse order of equipment, that is, pipeline stages s1 to s P From device G P Start allocating in stages until device G1, then stage s P+1 ~s 2P The reverse order starts from device G1 and allocates in stages until it returns to device G. P It forms an inverted V shape. Figure 4 As shown in the figure, taking the four-device pipeline as an example, stages s1 to s4 of the inverted V-shaped pipeline are respectively assigned to devices G4 to G1, and stages s5 to s8 are respectively assigned to devices G1 to G4. During training, the calculation of each micro-batch starts from device G4 and proceeds to device G1, and then continues from device G1 back to device G4.

[0044] S2. By integrating the V-shaped staggered pipeline and the inverted V-shaped staggered pipeline and adjusting the mapping, the devices responsible for the same pipeline stage are placed in the same server, and the two pipelines are scheduled to be located in the same stage of different devices at the same time, forming a bidirectional staggered pipeline parallel mode.

[0045] In the embodiment, by integrating the V-shaped staggered pipeline and the inverted V-shaped staggered pipeline, the device mapping is adjusted, and the devices responsible for the same pipeline stage (for example, devices G1 and G2) are mapped to the same pipeline stage. P , G2 and G P-1 , or G3 and G P-2 The device mapping relationship determines the communication bandwidth between devices. Devices in the same server are connected by high-speed networks, while devices in different servers are connected by low-speed networks. Figure 5As shown, the devices G1, G8, G3 and G6 responsible for stages s1 and s8, s3 and s6 are all located in server 1, and the devices G2, G7, G4 and G5 responsible for stages s2 and s7, s4 and s5 are all located in server 2, so that different devices responsible for the same stages are interconnected through a high-speed network.

[0046] After placing the devices responsible for the same pipeline stage in the same server, different micro-batches or the output results of the previous stage are input at the same time, and the V-shaped and inverted V-shaped staggered pipelines are scheduled in parallel to be located in the same stage of different devices, forming a bidirectional staggered pipeline parallel mode, such as Figure 4 As shown. Device G1 is responsible for stages s1 and s8 of the V-shaped pipeline and stages s4 and s5 of the inverted V-shaped pipeline. Device G4 is responsible for stages s4 and s5 of the V-shaped pipeline and stages s1 and s8 of the inverted V-shaped pipeline. At the first moment of training, stage s1 on devices G1 and G4 is scheduled to calculate micro-batches 1 and 3 respectively. At the second moment, stage s2 on devices G2 and G3 is scheduled to calculate micro-batches 1 and 3 respectively. This is repeated for subsequent training.

[0047] S3. Introduce an early gradient synchronization strategy to train the large model in a bidirectional staggered pipeline parallel mode. After the early synchronization conditions are met or the forward and reverse calculations of all micro-batches in each pipeline are completed, perform global gradient synchronization between devices storing the same pipeline stage, and then update the parameters of the large model to accelerate the training of the large model.

[0048] In the embodiment, an advance gradient synchronization strategy is introduced. Specifically, an advance synchronization strategy is introduced for global gradient synchronization to optimize the gradient synchronization, and the gradient synchronization operation of the sub-model M2 is advanced to between the reverse calculations of the last two micro-batches of the sub-model M1 on the device where it is located. The time when the global gradient synchronization ends is determined by the stage at which the gradient synchronization of the sub-model M1 is last completed. These stages are generally determined by the pipeline starting device (for example, device G1 or G2). P ) is responsible for training the large model according to the optimized bidirectional interleaved pipeline parallel mode. After the advance synchronization condition is met or the forward and reverse calculations of all micro-batches in each pipeline are completed, the devices responsible for the same pipeline stage (for example, devices G1 and G P ) performs global gradient synchronization and then updates the parameters of the large model.

[0049] Among them, satisfying the early synchronization condition means that there are pipeline bubbles between the reverse calculations of the last two micro-batches of sub-model M1 on the device. These bubbles can be used to complete the gradient synchronization communication of sub-model M2 in advance, and accordingly, the gradient synchronization of sub-model M1 can be started immediately when the reverse calculation of the last micro-batch is completed. Figure 6As shown in the figure, there are bubbles between the reverse calculations of micro-batches 1 and 2 of sub-model M1 on device G1 or micro-batches 3 and 4 of sub-model M1 on device G4. Therefore, the gradient synchronization of stages s5 and s8 of sub-model M2 on devices G1 and G4 can be advanced to the reverse calculations of the last two micro-batches of sub-model M1 (micro-batches 1 and 2 of device G1 or micro-batches 3 and 4 of device G4) and the advanced gradient synchronization can be applied between them. After the reverse calculation of the last micro-batch (micro-batch 2 or 4) is completed, the gradient synchronization of stages s1 and s4 of sub-model M1 can be started immediately, so that each device can complete the final gradient synchronization closer to the same time.

[0050] In order to clearly demonstrate the bidirectional interleaved pipeline parallel method for accelerating large model training, a bidirectional interleaved pipeline parallel system for accelerating large model training is provided in the embodiment ( Figure 2 As shown) includes: a large model segmentation module, a V-shaped staggered scheduling module, an adjustment device mapping module, a bidirectional staggered pipeline parallel module, a communication optimization module and a training module;

[0051] The large model segmentation module is used to segment the large model by layer according to the normal training order in an interleaved manner, and divide it into a plurality of stages twice the number of training devices, so that these stages can be executed in parallel on multiple devices;

[0052] The V-shaped staggered scheduling module is used to distribute the model slices to each device in a V-shaped manner to form a pipeline stage;

[0053] The device mapping adjustment module is used to place devices responsible for the same pipeline stage in the same server to optimize the communication bandwidth between the stages;

[0054] The bidirectional interleaved pipeline parallel module is used to parallelly schedule V-shaped and inverted V-shaped interleaved pipelines located at the same stage of different devices, forming a bidirectional interleaved pipeline parallel mode, allowing each device to carry and process pipeline stages in multiple directions;

[0055] The communication optimization module is used to introduce a communication optimization strategy, use the high-speed network in the server to perform global gradient synchronization, and introduce an advance synchronization strategy for global gradient synchronization to optimize it, so that each device can complete the final gradient synchronization closer to the same time;

[0056] The training module is used to train the large model according to the optimized bidirectional interleaved pipeline parallel mode and update the parameters of the large model.

[0057] In an embodiment, a bidirectional interleaved pipeline parallel device for accelerating large model training is also provided, including a memory and a processor, the memory is used to store a computer program, and the processor is used to implement a bidirectional interleaved pipeline parallel method when executing the computer program.

[0058] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A bidirectional interleaved pipeline parallel method for accelerating large model training, characterized in that: The following steps are involved: Using distributed clusters, the large model is sequentially divided into model slices in an interleaved manner, and the model slices are distributed to each device in a V-shaped manner to form a pipeline stage. Each pipeline stage is scheduled sequentially to form a V-shaped interleaved pipeline, and the devices are reversed to form an inverted V-shaped interleaved pipeline. By integrating the V-shaped interleaved pipeline and the inverted V-shaped interleaved pipeline, and adjusting the mapping, the devices responsible for the same pipeline stage are placed in the same server, and the two pipelines are scheduled to be located in the same stage of different devices at the same time, forming a bidirectional interleaved pipeline parallel mode; An early gradient synchronization strategy is introduced to train the large model in a bidirectional staggered pipeline parallel mode. After the early synchronization conditions are met or the forward and backward calculations of all micro-batches in each pipeline are completed, global gradient synchronization is performed between devices storing the same pipeline stage, and then the parameters of the large model are updated to accelerate the training of the large model.

2. The bidirectional interleaved pipeline parallel method according to claim 1, characterized in that: The method of sequentially dividing the large model into model slices in an interlaced manner includes: According to the number of pipeline devices P, the large model to be trained is divided into 2P model slices, among which slices 1 to P are divided into the first group, becoming sub-model M1; slices P+1 to 2P are divided into the second group, becoming sub-model M2.

3. The bidirectional interleaved pipeline parallel method according to claim 1, characterized in that: The V-shaped staggered production line comprises: Distribute model slices to each device in a V-shaped manner to form pipeline stages s1 to s 2P , where stages s1 to s P Starting from device G1, the allocation is carried out in stages until device G P , then stage s P+1 to 2P Reverse the order from device G P The pipeline is allocated stage by stage until it returns to device G1, forming a V shape. The pipeline stages are scheduled sequentially to form a V-shaped staggered pipeline.

4. The bidirectional interleaved pipeline parallel method according to claim 1, characterized in that: The equipment is reversed to form an inverted V-shaped staggered production line, including: Reverse the order of the devices and distribute the model slices to each device in a V-shaped manner to form the pipeline stages s1 to s 2P , where stages s1 to s P From device G P Start allocating in stages until device G1, then stage s P+1 to 2P Reverse the order and allocate from device G1 stage by stage until device G P , forming an inverted V-shaped staggered assembly line.

5. The bidirectional interleaved pipeline parallel method according to claim 1, characterized in that: The simultaneous scheduling of two pipelines located at the same stage of different devices includes: Different micro-batches or the output results of the previous stage are input at the same time, and the V-shaped and inverted V-shaped staggered pipelines are scheduled in parallel at the same pipeline stage of different devices, forming a bidirectional staggered pipeline parallel mode.

6. The bidirectional interleaved pipeline parallel method according to claim 2, characterized in that: The introduction of the advance gradient synchronization strategy includes: optimizing the introduction of the advance synchronization strategy for global gradient synchronization, and advancing the gradient synchronization operation of the sub-model M2 to between the reverse calculations of the last two micro-batches of the sub-model M1 on the device.

7. The bidirectional interleaved pipeline parallel method according to claim 2, characterized in that: The condition for satisfying the early synchronization is that there are pipeline bubbles between the reverse calculations of the last two micro-batches of the sub-model M1 on the device.

8. A bidirectional interleaved pipeline parallel system for accelerating large model training, characterized in that: The system uses the bidirectional interleaved pipeline parallel method according to any one of claims 1 to 7 in training, and the bidirectional interleaved pipeline parallel system comprises: a large model segmentation module, a V-shaped interleaved scheduling module, an adjustment device mapping module, a bidirectional interleaved pipeline parallel module, a communication optimization module and a training module; The large model segmentation module is used to segment the large model by layer according to the normal training order in an interleaved manner, and divide it into a plurality of stages twice the number of training devices, so that these stages can be executed in parallel on multiple devices; The V-shaped staggered scheduling module is used to distribute the model slices to each device in a V-shaped manner to form a pipeline stage; The device mapping adjustment module is used to place devices responsible for the same pipeline stage in the same server to optimize the communication bandwidth between the stages; The bidirectional interleaved pipeline parallel module is used to parallelly schedule V-shaped and inverted V-shaped interleaved pipelines located at the same stage of different devices, forming a bidirectional interleaved pipeline parallel mode, allowing each device to carry and process pipeline stages in multiple directions; The communication optimization module is used to introduce a communication optimization strategy, use the high-speed network in the server to perform global gradient synchronization, and introduce an advance synchronization strategy for global gradient synchronization to optimize it, so that each device can complete the final gradient synchronization closer to the same time; The training module is used to train the large model according to the optimized bidirectional interleaved pipeline parallel mode and update the parameters of the large model.

9. A bidirectional interleaved pipeline parallel device for accelerating large model training, comprising a memory and a processor, wherein the memory is used to store a computer program, characterized in that: The processor is used to implement the bidirectional interleaved pipeline parallel method according to any one of claims 1 to 7 when executing the computer program.

Images