An asynchronous multi-directional pipeline parallel method to accelerate large model training

Abstract

The present invention discloses an asynchronous multi-directional pipeline parallel method for accelerating large model training, including: creating multiple pipeline parallel groups, with different device ordering between groups; allocating model slices of the large model to be trained to each device in each group in order, and each group contains a complete large model to be trained; selecting any group as an optimizer registration parallel group and registering an optimizer for the model slices of each device therein; training the large model to be trained according to the asynchronous multi-directional pipeline scheduling method, each device accumulates the gradients of the same model slice on each micro-data batch, and when the accumulated micro-data batches form a data batch, immediately or at the pipeline cavitation position, enters the next step; each device aggregates the accumulated gradients to the devices of the optimizer registration parallel group and uses the aggregated gradients to update parameters. The method of the present invention can take into account the convergence of training while ensuring high throughput, and reduce storage overhead without increasing additional communication overhead.

Description

Technical Field

[0001] The present invention belongs to the technical field of distributed parallel training, and in particular relates to an asynchronous multi-directional pipeline parallel method for accelerating large model training. Background Art

[0002] In recent years, large models have achieved widespread success in various fields by expanding the model size. However, training a large model is challenging. For example, to train a GPT3 model with 175 billion parameters, at 16-bit precision, it takes 300 GiB of space to store its model parameters alone, and it takes about 288 years to complete the training using a single V100 graphics card. Therefore, parallel training on a distributed cluster is very necessary.

[0003] By splitting the model between layers and distributing them to different devices, inter-layer model parallelism has now been widely used to train models that cannot be loaded on a single device. However, due to the sequential execution characteristics of forward and backward calculations in deep learning, naive inter-layer model parallelism will cause the cluster's devices to be idle for a large amount of time during training. To solve the above problem, pipeline parallelism splits a single data batch into multiple micro-data batches and injects these micro-data batches into the pipeline at one time. This approach ensures that multiple devices are active in the pipeline at the same time, thereby reducing or even eliminating the time that the device is idle.

[0004] According to the parameter update strategy, pipeline parallelism can be divided into synchronous pipeline parallelism and asynchronous pipeline parallelism. In the synchronous pipeline, the gradients of all micro-data batches in the same data batch will be accumulated. After the backward calculation of all micro-data batches in the current data batch is completed, the accumulated gradients will be used to update the parameters, and then the next data batch will be read in and the above process will be repeated. Synchronous pipeline parallelism can ensure the convergence of training, but it cannot completely eliminate the device idle time in the inter-layer model parallelism. These times are called pipeline bubbles in pipeline parallelism. Asynchronous pipeline continuously injects micro-data batches into the pipeline and performs a parameter update after the backward calculation of each micro-data is completed. In the stable operation stage, there are no pipeline bubbles in the asynchronous pipeline. However, the asynchronous update strategy will cause the problem of delayed gradient and parameter mismatch, thereby destroying the convergence of training. In addition, the commonly used scheduling strategy in asynchronous pipeline parallelism is the 1F1B strategy, which means that in the stable stage of the pipeline, each device alternately performs forward and backward calculations. Under this strategy, the front devices in the pipeline have many other microdata batch calculations between their forward and backward calculations of the same microdata batch, which causes the device to store activation values ​​of multiple microdata batches. However, the back devices have fewer or even no other microdata batch calculations between their forward and backward calculations of the same microdata batch, so fewer activation values ​​are stored, resulting in inconsistent storage overhead between devices.

[0005] In summary, current pipeline parallel methods usually cannot take into account both high throughput and normal convergence, and commonly used pipeline scheduling strategies have the problem of unbalanced equipment overhead. Therefore, it is necessary to develop new pipeline parallel methods to meet the needs of large-scale model training. Summary of the invention

[0006] In view of the above, the purpose of the present invention is to provide an asynchronous multi-directional pipeline parallel method for accelerating large model training. Multi-directional pipeline scheduling is used in the asynchronous update strategy to solve the problem of unbalanced video memory overhead in traditional asynchronous pipeline parallelism and the problem of being sensitive to model uniformity. While improving the model training throughput, it can also ensure the convergence of training. In addition, the present invention also introduces zero-redundancy optimizer technology, which reduces the video memory overhead occupied by the optimizer state by selecting a parallel group and registering the optimizer for the model slices allocated to each device therein, and has broad application prospects in the fields of distributed parallel training and large model training.

[0007] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows:

[0008] An asynchronous multi-directional pipeline parallel method for accelerating large model training provided by an embodiment of the present invention includes the following steps:

[0009] Create multiple parallel pipeline groups, each group contains all devices in the server, the device ordering between groups is different, and the number of groups is the same as the number of set pipeline directions;

[0010] The large model to be trained is evenly divided into layers to obtain model slices, and the model slices are sequentially allocated to each device in each group so that each group contains a complete large model to be trained;

[0011] Select any group as the optimizer registration parallel group, and register the optimizer for the model slices allocated to each device in the optimizer registration parallel group;

[0012] The large model is trained according to the asynchronous multi-directional pipeline scheduling method. Each device accumulates the gradients of the same model slice on each micro-data batch. When the accumulated micro-data batches can form a data batch, if there is no pipeline bubble in the pipeline, it will immediately enter the next step, otherwise it will enter the next step at the next position where the pipeline bubble appears;

[0013] Each device aggregates the gradients of the accumulated model slices on a data batch to the devices of the optimizer registered parallel group and uses the aggregated gradients to update parameters.

[0014] Preferably, the creating of multiple pipeline parallel groups, each group including all devices in the server, the ordering of devices between groups is different, and the number of groups is the same as the number of set pipeline directions, includes:

[0015] create pipeline parallel groups, For the set number of pipeline directions, all devices participating in the training in the server are merged into each parallel group according to the order of the pipeline running direction in each group. Each group contains all the devices, and the arrangement order of the devices between groups is different.

[0016] Preferably, the set number of pipeline directions is no higher than half of the number of all devices in the server.

[0017] Preferably, the large model to be trained is evenly divided between layers to obtain model slices, and the model slices are sequentially allocated to each device in each group so that each group contains a complete large model to be trained, including:

[0018] The large model to be trained is evenly divided into layers according to the computing time of each network layer. share, is the number of all devices in the server, The model slices are allocated to the devices in each parallel group according to the arrangement order of the devices in the group. The sequence number of the model slice allocated to each device is the same as the sequence number of the device in each parallel group, so that each group finally contains a complete copy of the large model to be trained.

[0019] Preferably, the selecting any group as the optimizer registration parallel group, and registering the optimizer with the model slices allocated to each device in the optimizer registration parallel group, comprises:

[0020] Randomly select one parallel group as the optimizer registration parallel group, register the optimizer for the model slices allocated to each device in the optimizer registration parallel group, and do not register optimizers in other groups.

[0021] Preferably, before training the large model to be trained, first prepare a training data set and divide the training data set into Molecular datasets, is the number of pipeline parallel groups. Each parallel group takes a sub-dataset and batches it according to the size of the micro-data batch as input in the training process.

[0022] Preferably, the number of micro data batches in each data batch is not less than the number of devices in each group.

[0023] Preferably, the training of the large model to be trained in an asynchronous multi-directional pipeline scheduling manner includes:

[0024] After each parallel group injects all the acquired microdata batches into the pipeline, each group executes different pipeline directions according to the device order. In each parallel group, after the first device on the pipeline completes the forward calculation of a microdata batch, it passes the obtained activation value to the next device in the parallel group. The device that receives the activation value uses the activation value to perform forward calculation and passes the calculation result to the next device, and so on until the last device on the pipeline completes the forward calculation of a microdata batch, and then performs backward calculation. The backward calculation transfers the gradient in the opposite direction until the first device on the pipeline completes the backward calculation. For all acquired microdata batches, forward calculation and backward calculation are performed repeatedly in the above method.

[0025] Preferably, each device accumulates the gradients of the same model slice on each micro-data batch, and when the accumulated micro-data batches can form a data batch, if there is no pipeline cavitation in the pipeline, it immediately enters the next step, otherwise it enters the next step at the position where the pipeline cavitation appears next, including:

[0026] On each device, the gradients of the same model slice assigned to it on different microdata batches are accumulated. If there is no pipeline bubble in the pipeline, the next step is immediately entered for gradient aggregation and parameter update after the backward calculation of all microdata batches in a data batch is completed. If there is a pipeline bubble in the pipeline, the next step is entered for gradient aggregation and parameter update after the backward calculation of all microdata batches in a data batch is completed and the first pipeline bubble is encountered.

[0027] Preferably, each device aggregates the accumulated gradients of the model slices on a data batch to the devices of the optimizer registered parallel group and uses the aggregated gradients to perform parameter updates, including:

[0028] Each device in all parallel groups aggregates the gradients of the model slices allocated to it on a data batch to the devices in the parallel group registered by the optimizer that are allocated the same model slices;

[0029] When updating parameters, each device in the parallel group registered by the optimizer inputs the aggregated gradient of each model slice into the corresponding optimizer registered by each model slice to update the parameters of the large model, and broadcasts the updated parameters to the devices in the remaining groups. This completes a round of iteration, and the iteration is repeated until the large model training is completed.

[0030] The present invention uses asynchronous multi-directional pipeline scheduling to train the model, and introduces a partial synchronization mechanism in the parameter update step, which can improve the training throughput and training convergence. Compared with the prior art, the present invention has at least the following beneficial effects:

[0031] (1) The present invention introduces a multi-directional pipeline scheduling strategy based on the asynchronous pipeline parallel method, which can avoid the situation where the storage load of a single device is significantly higher than that of other devices, and at the same time reduce the impact of uneven model segmentation, thereby improving the training performance of large models;

[0032] (2) The present invention accumulates the gradients of the same model slice in each micro-data batch, and utilizes the obvious boundary between the forward and backward calculations of multiple micro-data batches in multi-directional scheduling to uniformly update parameters at the boundary between the forward and backward calculations of multiple micro-data batches. This can not only cover the pipeline bubbles at the boundary, but also achieve the overlap of calculation and communication, improve resource utilization, reduce communication overhead, and ensure high throughput while taking into account the convergence of training;

[0033] (3) The present invention introduces zero-redundancy optimizer technology. By selecting a parallel group as the optimizer registration parallel group and registering the optimizer for the model slices allocated to each device in it, each device only retains the optimizer state of one model slice, which can reduce storage overhead without increasing additional communication overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 It is a schematic diagram of the overall flow of an asynchronous multi-directional pipeline parallel method for accelerating large model training provided by an embodiment of the present invention;

[0036] Figure 2 is a schematic diagram of a pipeline parallel group provided by an embodiment of the present invention;

[0037] Figure 3 is a schematic diagram of model slice allocation provided by an embodiment of the present invention;

[0038] Figure 4 It is a schematic diagram of asynchronous multi-directional pipeline parallel scheduling provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0040] In view of the fact that pipeline parallel methods in the prior art usually cannot take into account both high throughput and normal convergence, and the common pipeline scheduling strategies have the problem of unbalanced device overhead, an embodiment of the present invention provides an asynchronous multi-directional pipeline parallel method for accelerating large model training: first, multiple pipeline parallel groups are created, each group contains all devices in the server, the device ordering between groups is different, and the number of groups is the same as the set number of pipeline directions; the large model to be trained is evenly divided between layers to obtain model slices, and the model slices are sequentially allocated to each device in each group, so that each group contains a complete large model to be trained; third, any group is selected as The optimizer registers a parallel group and registers the optimizer for the model slices allocated to each device in the optimizer registration parallel group; then the large model is trained according to the asynchronous multi-directional pipeline scheduling method, and each device accumulates the gradients of the same model slice on each micro-data batch. When the accumulated micro-data batches can form a data batch, if there is no pipeline bubble in the pipeline, it will immediately enter the next step, otherwise it will enter the next step at the next position where a pipeline bubble appears; finally, each device aggregates the gradients of the accumulated model slices on a data batch to the devices in the optimizer registration parallel group and uses the aggregated gradients to update the parameters.

[0041] Figure 1 FIG. 1 is a schematic diagram of the overall flow of an asynchronous multi-directional pipeline parallel method for accelerating large model training provided by an embodiment of the present invention. Figure 1 As shown, the embodiment provides an asynchronous multi-directional pipeline parallel method for accelerating large model training, including the following steps:

[0042] Step 1: Create pipeline parallel groups (parallel groups are numbered 0, 1, ..., -1), The number of pipeline directions set, usually set to the number of devices participating in the training ( All devices participating in the training in the server are merged into each parallel group according to the direction of the pipeline operation in each group. Each group contains all the devices, and the arrangement order of the devices between groups is different.

[0043] In this step, if Figure 2As shown, the number of devices participating in the training in the server is 8 (device 0, device 1, device 2, device 3, device 4, device 5, device 6, device 7), and the number of parallel groups is 4 (parallel group 0, parallel group 1, parallel group 2, parallel group 3). The arrangement order of the devices in each parallel group is different, and the pipeline direction in each parallel group is from device 0 to device 7 in the order of numbering, so each parallel group represents a pipeline direction. In the embodiment of the present invention, the arrangement order of the devices in each parallel group is the order of activation value transfer in the subsequent large model training process. Therefore, before implementing the method, the technician needs to arrange the pipeline in advance to ensure that the calculation of the same model slice in each parallel group can be executed in parallel. As for the traditional method, for example, if device 1 is responsible for slice 1 in all parallel groups, then after all parallel groups complete the forward calculation of slice 0 in parallel, the activation value needs to be sent to device 1. Device 1 needs to complete the forward calculation of all slices 1 and send the activation value to the device responsible for slice 2 in each parallel group. This will make device 1 a bottleneck in the pipeline, and the forward calculation of slices 1 and 2 cannot be executed in parallel, introducing a large number of pipeline bubbles. Therefore, this is not a good arrangement.

[0044] Step 2: Divide the large model to be trained evenly into copy, will The model slices are allocated to the devices in each parallel group according to the order in which the devices are arranged. Each device will receive model slices, with each group containing a copy of the full model.

[0045] In this step, if Figure 3 As shown in the figure, the large model to be trained is evenly divided into 8 parts (slice 0, slice 1, slice 2, slice 3, slice 4, slice 5, slice 6, slice 7), and each device is allocated 4 model slices. The sequence number of the slice is the same as the sequence number of the device in each parallel group. Take device 0 as an example. Figure 2 As shown, its serial numbers in parallel group 0, parallel group 1, parallel group 2, and parallel group 3 are 0, 3, 4, and 7 respectively. Therefore, the model slices allocated to device 0 are slices 0, 3, 4, and 7 respectively, and the same applies to the remaining devices.

[0046] Step 3: Use zero-redundancy optimizer technology, select any group as the optimizer registration parallel group, register the optimizer for the model slices allocated to each device on the optimizer registration parallel group, and do not register the remaining groups.

[0047] In this step, select parallel group 0 as the optimizer registration parallel group, such as Figure 3As shown, although there are 4 model slices on each device, each device will only register the optimizer for the model slice allocated to it in parallel group 0. For example, device 0 will only register the optimizer for slice 0, device 1 will only register the optimizer for slice 1, and so on. In the actual implementation process, any parallel group can be selected to register the optimizer. In order to meet the general habits of technicians and facilitate the explanation of steps, this article uses parallel group 0 as an example.

[0048] Step 4: Split the training dataset into Each parallel group takes a sub-dataset and batches the sub-datasets according to the size of the micro-data batch as input in the training process.

[0049] In this step, the training dataset is split into 4 sub-datasets, each of which contains several data batches, and each data batch contains 8 micro-data batches. The purpose of splitting the training dataset is to prevent each parallel group from reading the same data in each round of training. If the training dataset is not split, each round of iteration is equivalent to calculating the same data batch. Sub-gradients not only cause resource waste due to computational redundancy, but may also affect the convergence of model training in subsequent steps. A single data batch is further split into multiple micro-data batches in order to assign the computational tasks of a data batch to multiple devices, thereby achieving acceleration. In the specific implementation process, since each device directly reads the micro-data batch, the process of dividing the sub-data sets into data batches is skipped. Instead, the sub-data sets are directly divided into micro-data batches. When updating parameters, the accumulated gradient of the micro-data batch can be directly used to achieve the same result as using the gradient of the data batch for updating.

[0050] Step 5: Each parallel group obtains microdata batches from its respective sub-datasets, injects all microdata batches into the pipeline, and performs forward and backward calculations in an asynchronous multi-directional pipeline scheduling manner.

[0051] In this step, following the examples in steps 1 and 2, we can obtain Figure 4 The asynchronous multi-directional pipeline parallel scheduling diagram shown. Figure 4 The left side of Figure 2 The corresponding device sorting, the arrow direction indicates the pipeline direction in each parallel group, and the number in the ellipse indicates the sequence number of the assigned model slice. Figure 4In the table on the right side of , the left oblique solid line, right oblique solid line, left oblique dashed line, and right oblique dashed line in the four legends representing forward calculations represent the forward calculations of parallel group 0, parallel group 1, parallel group 2, and parallel group 3, respectively. The left oblique solid line, right oblique solid line, left oblique dashed line, and right oblique dashed line in the four legends representing backward calculations represent the backward calculations of parallel group 0, parallel group 1, parallel group 2, and parallel group 3, respectively. The numbers on each oblique line in the table represent the sequence number of the processed microdata batch, the blank grid represents the pipeline bubble, and the blank grid with the mark U indicates that gradient aggregation and parameter update are performed at the pipeline bubble. The device with sequence number 0 in each parallel group reads the microdata batch from the sub-data set obtained by the parallel group. After the forward calculation of a microdata batch is completed, the obtained activation value is transmitted to the next device in the parallel group according to the pipeline direction. The device that receives the activation value uses the activation value to perform forward calculation and transmits the calculation result to the next device, and so on until the forward calculation of a microdata batch is completed. The backward calculation passes the gradient in the opposite direction until the first device (device 0) in the pipeline completes the backward calculation, and then continues to select the next micro-data batch, repeating the above rules.

[0052] Step 6: After completing the backward calculation of each microdata batch, each device accumulates the gradients of the same model slice on different microdata batches. When the accumulated microdata batches can form a data batch, if there is no pipeline cavitation in the pipeline, immediately enter step 7, otherwise enter step 7 at the next position where a pipeline cavitation occurs.

[0053] In this step, if Figure 4 As shown in the figure, during the calculation process of every 8 micro-data batches, pipeline bubbles will appear at the junction of the forward and backward calculations. After the backward calculation of each 8 micro-data batches is completed, step 7 will be entered at the next pipeline bubble. In the actual implementation process, when or Different from the example setting, the position where the pipeline bubble appears will also change. At this time, after the backward calculation of all micro-data batches in a data batch is completed, the next step is entered when the first pipeline bubble is encountered. If there is no pipeline bubble in the pipeline, the next step is immediately entered when the backward calculation of all micro-data batches in a data batch is completed. Different from the practice of performing a parameter update once the backward calculation of each micro-data batch is completed in other asynchronous pipeline parallel, this method accumulates the gradients of all micro-data batches belonging to the same data batch, and enters the parameter update step when the gradient calculation of all micro-data batches in a data batch is completed. The practice of updating parameters at the junction of forward and backward calculations allows the same micro-data batch to use parameters that differ by at most one update when calculating on different devices, and also allows the parameters used in the forward and backward calculations of the same micro-data batch on the same device to differ by only one update, thereby improving the convergence of training.

[0054] Step 7: Aggregate the gradients of the model slices in each parallel group to the devices of the parallel group registered by the optimizer. The devices of the parallel group registered by the optimizer use the aggregated gradients to update the parameters and broadcast the updated parameters to the devices of the remaining parallel groups.

[0055] In this step, when performing gradient aggregation, the gradients of the model slices allocated to each device in all parallel groups are aggregated to the devices that have the same model slice in parallel group 0 (the optimizer registration parallel group). For example, when following Figure 2 , Figure 3 In the above step settings, device 0, device 3, device 4, and device 7 will aggregate the gradients of their slice 3 to device 3 in parallel group 0, because the slice allocated to device 3 in parallel group 0 is slice 3, and the rest of the slices are analogous. After the gradient aggregation is completed, the devices in parallel group 0 will use the optimizer registered in step 3 to update the parameters of the large model, and broadcast the updated parameters to the devices in the rest of the parallel groups. At this point, one round of iteration is completed, and steps 5, 6, and 7 are repeated until the training is completed.

[0056] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An asynchronous multi-directional pipeline parallel method for accelerating large model training, characterized in that: The following steps are involved: Create multiple parallel pipeline groups, each group contains all devices in the server, the device ordering between groups is different, and the number of groups is the same as the number of set pipeline directions; The large model to be trained is evenly divided into layers to obtain model slices, and the model slices are sequentially allocated to each device in each group so that each group contains a complete large model to be trained; Select any group as the optimizer registration parallel group, and register the optimizer for the model slices allocated to each device in the optimizer registration parallel group; The large model is trained according to the asynchronous multi-directional pipeline scheduling method. Each device accumulates the gradients of the same model slice on each micro-data batch. When the accumulated micro-data batches can form a data batch, if there is no pipeline bubble in the pipeline, it will immediately enter the next step, otherwise it will enter the next step at the next position where the pipeline bubble appears; Each device aggregates the gradients of the accumulated model slices on a data batch to the devices of the optimizer registered parallel group and uses the aggregated gradients to update parameters.

2. The asynchronous multi-directional pipeline parallel method for accelerating large model training according to claim 1, characterized in that: The method creates multiple parallel pipeline groups, each of which contains all devices in the server. The device ordering between groups is different. The number of groups is the same as the number of set pipeline directions, including: create pipeline parallel groups, For the set number of pipeline directions, all devices participating in the training in the server are merged into each parallel group according to the order of the pipeline running direction in each group. Each group contains all the devices, and the arrangement order of the devices between groups is different.

3. The asynchronous multi-directional pipeline parallel method for accelerating large model training according to claim 1, characterized in that: The number of pipeline directions is set to no more than half of the number of all devices in the server.

4. The asynchronous multi-directional pipeline parallel method for accelerating large model training according to claim 1, characterized in that: The large model to be trained is evenly divided between layers to obtain model slices, and the model slices are sequentially allocated to each device in each group so that each group contains a complete large model to be trained, including: The large model to be trained is evenly divided into share, is the number of all devices in the server, The model slices are allocated to the devices in each parallel group according to the arrangement order of the devices in the group. The serial number of the model slice allocated to each device is the same as the serial number of the device in each parallel group, so that each group finally contains a complete copy of the large model to be trained.

5. The asynchronous multi-directional pipeline parallel method for accelerating large model training according to claim 1, characterized in that: The selecting any group as the optimizer registration parallel group, and registering the optimizer for the model slices allocated to each device in the optimizer registration parallel group, includes: Randomly select one parallel group as the optimizer registration parallel group, register the optimizer for the model slices allocated to each device in the optimizer registration parallel group, and do not register optimizers in other groups.

6. The asynchronous multi-directional pipeline parallel method for accelerating large model training according to claim 1, characterized in that: Before training the large model, first prepare the training data set and divide the training data set into Molecular datasets, is the number of pipeline parallel groups. Each parallel group takes a sub-dataset and batches it according to the size of the micro-data batch as input in the training process.

7. The asynchronous multi-directional pipeline parallel method for accelerating large model training according to claim 6, characterized in that: The number of micro-data batches in each data batch shall not be less than the number of devices in each group.

8. The asynchronous multi-directional pipeline parallel method for accelerating large model training according to claim 1, characterized in that: The method of training the large model according to the asynchronous multi-directional pipeline scheduling method includes: After each parallel group injects all the acquired microdata batches into the pipeline, each group executes different pipeline directions according to the device order. In each parallel group, after the first device on the pipeline completes the forward calculation of a microdata batch, it passes the obtained activation value to the next device in the parallel group. The device that receives the activation value uses the activation value to perform forward calculation and passes the calculation result to the next device, and so on until the last device on the pipeline completes the forward calculation of a microdata batch, and then performs backward calculation. The backward calculation transfers the gradient in the opposite direction until the first device on the pipeline completes the backward calculation. For all acquired microdata batches, forward calculation and backward calculation are performed repeatedly in the above method.

9. The asynchronous multi-directional pipeline parallel method for accelerating large model training according to claim 8, characterized in that: Each device accumulates the gradients of the same model slice on each micro-data batch, and when the accumulated micro-data batches can form a data batch, if there is no pipeline cavitation in the pipeline, it immediately enters the next step, otherwise it enters the next step at the position where the pipeline cavitation appears next, including: On each device, the gradients of the same model slice assigned to it on different microdata batches are accumulated. If there is no pipeline bubble in the pipeline, the next step is immediately entered for gradient aggregation and parameter update after the backward calculation of all microdata batches in a data batch is completed. If there is a pipeline bubble in the pipeline, the next step is entered for gradient aggregation and parameter update after the backward calculation of all microdata batches in a data batch is completed and the first pipeline bubble is encountered.

10. The asynchronous multi-directional pipeline parallel method for accelerating large model training according to claim 1, characterized in that: Each device aggregates the accumulated gradients of the model slices on a data batch to the device of the optimizer registered parallel group and uses the aggregated gradients to update parameters, including: Each device in all parallel groups aggregates the gradients of the model slices allocated to it on a data batch to the devices in the parallel group registered by the optimizer that are allocated the same model slices; When updating parameters, each device in the parallel group registered by the optimizer inputs the aggregated gradient of each model slice into the corresponding optimizer registered by each model slice to update the parameters of the large model, and broadcasts the updated parameters to the devices in the remaining groups. This completes a round of iteration, and the iteration is repeated until the large model training is completed.

Images