Model training method, model training device, electronic equipment and medium

Abstract

This application relates to the field of model training technology, and discloses a model training method, a model training device, an electronic device, and a medium. The model training method includes: dividing the model to be trained into multiple computational stages along the network depth direction, and allocating these stages to multiple computing units in a distributed computing cluster; dividing each mini-batch of training data into multiple micro-batches; and sequentially performing forward propagation and backward propagation computations on the micro-batches based on the multiple computational stages and micro-batches; the backward propagation computation steps include activation value gradient calculation and weight gradient calculation; wherein, the weight gradient calculation is initiated after obtaining the corresponding activation value gradient results without waiting for all activation value gradient calculations to be completed. According to this model training method, the utilization rate of distributed computing resources can be improved.

Description

Technical Field

[0001] This application relates to the field of model training technology, and in particular to a model training method, model training device, electronic device and medium. Background Technology

[0002] In recent years, the field of artificial intelligence has developed rapidly, and the number of parameters in large language models has reached trillions. The training process places extremely high demands on computing and memory resources. Distributed training technology, by distributing computing tasks among multiple computing units to work collaboratively, has become the core infrastructure supporting the training of large models.

[0003] Currently, the mainstream distributed training parallel strategy is pipeline parallelism. However, in related technologies, model training methods using pipeline parallelism suffer from numerous pipeline bubbles. Pipeline bubbles refer to the phenomenon where, during the filling and emptying phases of the pipeline, most computing units remain idle due to dependencies, resulting in decreased utilization of computing resources. Summary of the Invention

[0004] This application provides a model training method, a model training device, an electronic device, and a medium to reduce pipeline bubbles in the parallel training process and improve the utilization of distributed computing resources.

[0005] In a first aspect, embodiments of this application provide a model training method, comprising: dividing the model to be trained into multiple computational stages along the network depth direction, and allocating the multiple computational stages to multiple computing units of a distributed computing cluster; dividing each mini-batch of training data into multiple micro-batches; and sequentially performing forward propagation computation and backward propagation computation on the micro-batches based on the multiple computational stages and the micro-batches; in the forward propagation computation step, the multiple micro-batches of each mini-batch sequentially pass through the multiple computing units in the forward direction of the network depth direction; the backward propagation computation step includes activation value gradient calculation and weight gradient calculation; in the activation... In the step of calculating the value gradient, the plurality of micro-batches of each mini-batch are sequentially passed through the plurality of computing units in the reverse direction of the network depth. In each computing stage, the activation value gradient from the next adjacent computing stage is received, and the activation value gradient of the current computing stage is calculated. In the step of calculating the weight gradient, in each computing stage, the weight gradient of the current computing stage is calculated based on the local gradient intermediate results obtained by the activation value gradient calculation step and the corresponding forward propagation activation values. The weight gradient calculation is initiated without waiting for the completion of all activation value gradient calculations, based on the activation value gradient results already obtained.

[0006] According to some of the foregoing embodiments of the first aspect of this application, the forward propagation computation includes: in each computation stage, receiving the activation value of the adjacent previous computation stage; calculating the linear output of the current computation stage based on the received activation value of the adjacent previous computation stage and the weight matrix of the current computation stage; generating the activation value of the current computation stage based on the linear output and the activation function; and outputting the activation value of the current computation stage to the adjacent next computation stage.

[0007] According to some of the foregoing embodiments of the first aspect of this application, the activation value gradient calculation includes: receiving the activation value gradient corresponding to the activation value output by the current calculation stage from the adjacent next calculation stage; generating a local gradient intermediate result of the current calculation stage based on the activation value gradient from the adjacent next calculation stage, the gradient calculation rule of the activation function, and the linear output obtained by the current calculation stage in the forward propagation calculation; calculating the activation value gradient corresponding to the input activation value of the current calculation stage based on the local gradient intermediate result and the weight matrix of the current calculation stage; and outputting the activation value gradient calculated by the current calculation stage to the adjacent previous calculation stage.

[0008] According to some of the foregoing embodiments of the first aspect of this application, the weight gradient calculation includes: in each calculation stage, calculating the weight gradient of the current calculation stage based on the intermediate results of the local gradient and the activation values ​​of the adjacent previous calculation stage.

[0009] According to some of the foregoing embodiments of the first aspect of this application, the model training method further includes: for two consecutive adjacent mini-batches, scheduling the computational tasks of the final stage of the preceding mini-batch to be executed in the idle time slice of the warm-up stage of the following mini-batch.

[0010] According to some embodiments of the first aspect of this application, the step of allocating the plurality of computing stages to a plurality of computing units in a distributed computing cluster includes: allocating the plurality of computing stages to a plurality of computing units in a distributed computing cluster. The step of scheduling the computational tasks of the final stage of the preceding mini-batch to an idle time slice during the warm-up stage of the following mini-batch for execution in two consecutive adjacent mini-batches includes: for two consecutive adjacent mini-batches, scheduling the computational tasks of the final stage of the preceding mini-batch to an idle time slice during the warm-up stage of the following mini-batch for execution in an idle time slice during the warm-up stage of the following mini-batch. The computational tasks with longer durations are scheduled to the warm-up phase of the mini-batch described later. Execution is performed during idle time slices within the specified duration, wherein the time window length of the final stage is [length missing]. The time window length of the preheating stage is , The duration of forward propagation computation for a single micro-batch. The number of computing units.

[0011] According to some of the foregoing embodiments of the first aspect of this application, the step of dividing the model to be trained into multiple computational stages along the network depth direction and allocating the multiple computational stages to multiple computing units of a distributed computing cluster includes: dividing the model to be trained into multiple computational stages along the network depth direction. Each calculation stage, and the... Each computation stage is assigned to a distributed computing cluster. The computational unit, wherein the first... The computing unit carries the first Each of the aforementioned computational stages From 1 to Integers.

[0012] According to some of the foregoing embodiments of the first aspect of this application, the step of dividing the model to be trained into multiple computational stages along the network depth direction and allocating the multiple computational stages to multiple computing units of a distributed computing cluster includes: dividing the model to be trained into multiple computational stages along the network depth direction. Each calculation stage, and the... Each computation stage is assigned to a distributed computing cluster. The computational unit, wherein the first... The computing unit carries the first The computational stage and the first Each of the aforementioned computational stages From 1 to Integers.

[0013] Secondly, embodiments of this application provide a model training apparatus, comprising: an allocation module for dividing the model to be trained into multiple computational stages along the network depth direction and allocating the multiple computational stages to multiple computing units of a distributed computing cluster; a training data processing module for dividing each mini-batch of training data into multiple micro-batches; and a training execution module for sequentially performing forward propagation computation and backward propagation computation on the micro-batches based on the multiple computational stages and the micro-batches, wherein the training execution module comprises: a forward propagation computation submodule for performing forward propagation computation, such that the multiple micro-batches of each mini-batch sequentially pass through the multiple computing units in the forward direction of the network depth direction; and a backward propagation computation submodule for performing backward propagation computation, wherein the backward propagation computation submodule comprises: The system includes an activation gradient calculation unit and a weight gradient calculation unit. The activation gradient calculation unit performs activation gradient calculation, causing multiple micro-batches of each mini-batch to pass through the multiple calculation units sequentially in the reverse direction of the network depth. In each calculation stage, it receives activation gradients from the next adjacent calculation stage and calculates the activation gradient for the current calculation stage. The weight gradient calculation unit performs weight gradient calculation, wherein in each calculation stage, it calculates the weight gradient for the current calculation stage based on the intermediate local gradient results obtained by the activation gradient calculation unit and the corresponding forward propagation activation values. The weight gradient calculation is initiated without waiting for the completion of all activation gradient calculations, based on the already obtained corresponding activation gradient results.

[0014] Thirdly, embodiments of this application provide an electronic device, which includes a memory and a processor. The memory is communicatively connected to the processor. The memory stores instructions, and the processor invokes the instructions in the memory to cause the electronic device to execute a model training method according to any of the foregoing embodiments of the first aspect of this application.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed by a processor, implement a model training method according to any of the foregoing embodiments of the first aspect of this application.

[0016] The model training method according to embodiments of this application includes: dividing the model to be trained into multiple computational stages along the network depth direction, and allocating the multiple computational stages to multiple computing units of a distributed computing cluster; dividing each mini-batch of training data into multiple micro-batches; and performing forward propagation computation and backward propagation computation sequentially on the micro-batches based on the multiple computational stages and micro-batches. During forward propagation computation, the multiple micro-batches of each mini-batch pass through multiple computing units sequentially in the forward direction of the network depth direction. The backward propagation computation includes activation value gradient calculation and weight gradient calculation. During activation value gradient calculation, the multiple micro-batches of each mini-batch pass through multiple computing units sequentially in the backward direction of the network depth direction. At each computational stage, the activation value gradient from the adjacent next computational stage is received, and the activation value gradient of the current computational stage is calculated. The activation value gradient is then propagated backward along the network depth direction. During weight gradient calculation, at each computational stage, the weight gradient of the current computational stage is calculated based on the intermediate local gradient results obtained from the activation value gradient calculation step and the corresponding forward propagation activation values. Weight gradient calculation is initiated after the corresponding activation value gradient results have been obtained, without waiting for all activation value gradient calculations to complete. Therefore, when performing calculations on each mini-batch, the traditional backpropagation calculation is split into two logically and temporally independent sub-steps: activation value gradient calculation and weight gradient calculation. Compared to the traditional method of encapsulating activation value gradient calculation and weight gradient calculation together, this embodiment reduces the waiting time between activation value gradient calculation and weight gradient calculation, thereby reducing the number of pipeline bubbles in the pipelined parallel training process and improving the utilization of distributed computing resources. On the other hand, when performing weight gradient calculation, it is based on the local gradient intermediate results and local forward propagation activation values ​​from the activation value gradient calculation process, eliminating the need for cross-computation unit communication, thus reducing communication overhead. Furthermore, for each calculation stage, the cache of the corresponding computation unit can be released immediately after obtaining the weight gradient, reducing peak cache requirements. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0018] Figure 1 This is a flowchart of one embodiment of the model training method according to this application; Figure 2 This is a timing diagram of a mini-batch pipeline in one embodiment of the model training method according to this application; Figure 3 A timing diagram of a mini-batch pipeline in a comparative example based on traditional model training methods; Figure 4 This is a timing diagram of the pipeline between two adjacent mini-batches in an alternative embodiment of the model training method according to this application; Figure 5 This is a timing diagram of the pipeline between two adjacent mini-batches in another alternative embodiment of the model training method according to this application; Figure 6 This is a schematic diagram of the structure of one embodiment of the model training apparatus according to this application; Figure 7 This is a schematic diagram of the structure of the training execution module in one embodiment of the model training apparatus according to this application; Figure 8 This is a schematic diagram of the hardware structure of an embodiment of the electronic device according to this application. Detailed Implementation

[0019] The technical solutions in the embodiments (or "implementations") of this application will be clearly and completely described herein with reference to the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.

[0020] If the embodiments of this application contain terms relating to directional indications or positional relationships (such as up, down, left, right, front, back, inside, outside, top, bottom, center, vertical, horizontal, longitudinal, transverse, length, width, counterclockwise, clockwise, axial, radial, circumferential, etc.), such terms are only used to explain the relative positional relationships and movement of the components in a specific posture (as shown in the attached figures); if the specific posture changes, the directional indications or positional relationships will also change accordingly. Furthermore, the terms "first" and "second" used in the embodiments of this application are only for descriptive convenience and should not be construed as indicating or implying relative importance.

[0021] This application provides a model training method. Figure 1 This is a flowchart of one embodiment of the model training method according to this application. The model training method includes steps S110 to S130.

[0022] In step S110, the model to be trained is divided into multiple computation stages along the network depth direction, and the multiple computation stages are assigned to multiple computation units of the distributed computing cluster.

[0023] In some embodiments, the computing unit is a graphics processing unit (GPU). In some embodiments, the distributed computing cluster is a computing cluster that includes multiple graphics processing units.

[0024] In step S120, each mini-batch of training data is divided into multiple micro-batches.

[0025] In step S130, based on multiple computation stages and micro-batches, forward propagation computation and backward propagation computation are performed sequentially on the micro-batches.

[0026] In the forward propagation computation step, multiple micro-batches of each mini-batch are sequentially passed through multiple computation units in the forward direction of the network depth.

[0027] In some embodiments, forward propagation computation includes: at each computation stage, receiving activation values ​​from the adjacent previous computation stage; calculating the linear output of the current computation stage based on the received activation values ​​from the adjacent previous computation stage and the weight matrix of the current computation stage; generating activation values ​​for the current computation stage based on the linear output and the activation function; and outputting the activation values ​​of the current computation stage to the adjacent next computation stage.

[0028] In this embodiment, the preceding and following computation stages are determined according to the network depth direction. For example, if the model to be trained is divided into four computation stages along the network depth direction, then these include a first computation stage (Stage-1), a second computation stage (Stage-2), a third computation stage (Stage-3), and a fourth computation stage (Stage-4) arranged along the network depth direction. Here, the first computation stage (Stage-1) is the adjacent preceding computation stage of the second computation stage (Stage-2), and the third computation stage (Stage-3) is the adjacent following computation stage of the second computation stage (Stage-2).

[0029] In some embodiments, the current computing stage is Stage- The weight matrix is Receive from the adjacent previous computation stage - activation value Based on the received activation value from the adjacent previous computation stage The weight matrix at the current calculation stage The linear output of the current calculation stage is calculated. The linear output of the current calculation stage can be obtained according to the following formula. : .

[0030] Linear output based on the current computation stage Activation function Generate the activation value for the current computation phase. The activation value for the current computation stage can be obtained using the following formula. : .

[0031] The backpropagation calculation steps include calculating the activation gradient and the weight gradient.

[0032] In the step of calculating activation value gradient, multiple micro-batches of each mini-batch are sequentially passed through multiple computing units in the reverse direction of network depth. At each computing stage, activation value gradients from the next adjacent computing stage are received, and the activation value gradient of the current computing stage is calculated.

[0033] In this embodiment, the loss is Activation value gradient It is a loss. relative to the input activation value in the current computation phase The gradient.

[0034] In the weight gradient calculation step, at each calculation stage, the weight gradient of the current calculation stage is calculated based on the intermediate local gradient results obtained from the activation value gradient calculation step and the corresponding forward propagation activation values.

[0035] In this embodiment, the weight gradient It is a loss. The trainable parameters, i.e., the weight matrix, are relative to the current layer's computational stage. The gradient.

[0036] In this embodiment, the weight gradient calculation is initiated without waiting for all activation value gradient calculations to be completed, based on the already obtained corresponding activation value gradient results.

[0037] In some embodiments, activation value gradient calculation includes: receiving activation value gradients from the next adjacent calculation stage that correspond to the activation values ​​output by the current calculation stage; generating intermediate local gradient results for the current calculation stage based on the activation value gradients from the next adjacent calculation stage, the gradient calculation rules of the activation function, and the linear output obtained by the current calculation stage in the forward propagation calculation; calculating the activation value gradient corresponding to the input activation values ​​of the current calculation stage based on the intermediate local gradient results and the weight matrix of the current calculation stage; and outputting the activation value gradient calculated by the current calculation stage to the previous adjacent calculation stage.

[0038] In some embodiments, receive from the adjacent next computation stage - The activation value output in the current calculation phase. Corresponding activation gradient Based on the next computational stage - Activation gradient Activation function The gradient calculation rules and the linear output obtained in the forward propagation calculation at the current calculation stage. Generate intermediate local gradient results for the current computation stage. The intermediate local gradient result for the current calculation stage can be obtained according to the following formula. : ; In the above formula, the symbol This represents element-wise multiplication. It is an activation function The derivative of .

[0039] In some embodiments, based on the current local gradient intermediate results The weight matrix at the current calculation stage The calculated activation value is the same as the input value in the current calculation stage. Corresponding activation gradient The activation value can be obtained from the input during the current calculation phase using the following formula. Corresponding activation gradient : .

[0040] In some embodiments, weight gradient calculation includes: at each calculation stage, calculating the weight gradient of the current calculation stage based on the intermediate results of the local gradient and the activation values ​​of the adjacent previous calculation stage.

[0041] In some embodiments, based on local gradient intermediate results Activation value of the adjacent previous calculation stage The weight gradient at the current computation stage is calculated. The weight gradient for the current computation stage can be obtained using the following formula. : .

[0042] In this embodiment, the model to be trained is divided into multiple computational stages along the network depth direction, and these multiple computational stages are allocated to multiple computing units in a distributed computing cluster. This includes: dividing the model to be trained into multiple computational stages along the network depth direction. Each computational stage, and will Each computation stage is assigned to a distributed computing cluster. The computational unit, in which the first The computing unit carries the first Each calculation stage, in which... It is a positive integer. From 1 to Integers.

[0043] Figure 2 This is a timing diagram of a mini-batch pipeline in one embodiment of the model training method according to this application. Figure 2 This illustrates a pipeline for a mini-batch MP1. In this embodiment, the distributed computing cluster includes four computing units: GPU1, GPU2, GPU3, and GPU4. The model to be trained is divided into four computational stages along the network depth direction, and these stages are assigned to the four computing units of the distributed computing cluster. The forward floating-point operations (FLOPs) of each computational stage are approximately equal, and the error threshold of FLOPs is... .like Figure 2 In this embodiment, each mini-batch of training data is divided into 4 micro-batches. The forward propagation computation time for a single micro-batch is... .like Figure 2 In this embodiment, the pipeline duration bubble rate , The number of computational units represents the pipeline parallelism. This refers to the quantity of the micro-batch.

[0044] Figure 3 A timing diagram of a mini-batch pipeline in a comparative example of traditional model training methods. Figure 3 This illustrates a production line for a mini-batch of MP1s. The comparison is the same as the one described above. Figure 2 The number of computing units and the number of micro-batches are the same in the embodiments involved, that is, the number of computing units. Also 4, the number of micro-lots in each mini-lot. The value is 4. In the comparative example, the traditional pipelined parallel model training method is adopted, that is, for each mini-batch, forward propagation and back propagation calculations are performed. In the back propagation calculation, the weight gradient calculation and activation value gradient calculation are encapsulated together. Therefore, the time for a single mini-batch to perform forward propagation calculation is... At that time, the time for a single micro-batch to perform backpropagation computation is 2. .like Figure 3 In the comparative example, the production line duration bubble rate .

[0045] according to Figure 2 and Figure 3 Using the model training method of this application embodiment, the bubble rate can be reduced by 23 percentage points under the same conditions of the number of computing units and the number of micro-batches.

[0046] The model training method according to embodiments of this application includes: dividing the model to be trained into multiple computational stages along the network depth direction, and allocating the multiple computational stages to multiple computing units of a distributed computing cluster; dividing each mini-batch of training data into multiple micro-batches; and performing forward propagation computation and backward propagation computation sequentially on the micro-batches based on the multiple computational stages and micro-batches. During forward propagation computation, the multiple micro-batches of each mini-batch pass through multiple computing units sequentially in the forward direction of the network depth direction. The backward propagation computation includes activation value gradient calculation and weight gradient calculation. During activation value gradient calculation, the multiple micro-batches of each mini-batch pass through multiple computing units sequentially in the backward direction of the network depth direction. At each computational stage, the activation value gradient from the adjacent next computational stage is received, and the activation value gradient of the current computational stage is calculated. The activation value gradient is then propagated backward along the network depth direction. During weight gradient calculation, at each computational stage, the weight gradient of the current computational stage is calculated based on the intermediate local gradient results obtained from the activation value gradient calculation step and the corresponding forward propagation activation values. Weight gradient calculation is initiated after the corresponding activation value gradient results have been obtained, without waiting for all activation value gradient calculations to complete. Therefore, when performing calculations on each mini-batch, the traditional backpropagation calculation is split into two logically and temporally independent sub-steps: activation value gradient calculation and weight gradient calculation. Compared to the traditional method of encapsulating activation value gradient calculation and weight gradient calculation together, this embodiment reduces the waiting time between activation value gradient calculation and weight gradient calculation, thereby reducing the number of pipeline bubbles in the pipelined parallel training process and improving the utilization of distributed computing resources. On the other hand, when performing weight gradient calculation, it is based on the intermediate local gradient results and local forward propagation activation values ​​from the activation value gradient calculation process, eliminating the need for cross-computation unit communication, thus reducing communication overhead. Furthermore, for each calculation stage, the cache usage of the corresponding computing unit can be released immediately after obtaining the weight gradient, such as releasing GPU memory usage, reducing peak cache requirements.

[0047] Figure 4 This is a timing diagram of the pipeline between two adjacent mini-batches in an alternative embodiment of the model training method according to this application. Figure 4 In the alternative embodiments involved, some steps of the model training method are the same as... Figure 2 The steps involved in the embodiments are similar, and the following will mainly describe the differences between the alternative embodiments and the aforementioned embodiment.

[0048] In this embodiment, the model to be trained is divided into multiple computational stages along the network depth direction, and these multiple computational stages are allocated to multiple computing units in a distributed computing cluster. This includes: dividing the model to be trained into multiple computational stages along the network depth direction. Each computational stage, and will Each computation stage is assigned to a distributed computing cluster. The computational unit, in which the first The computing unit carries the first Each calculation stage, in which... It is a positive integer. From 1 to Integers.

[0049] In some embodiments, the model training method further includes: for two consecutive adjacent mini-batches, scheduling the computational tasks of the final stage of the previous mini-batch to be executed in the idle time slices of the warm-up stage of the subsequent mini-batch, so as to reduce the idle time of the computational units in the warm-up stage of the subsequent mini-batch.

[0050] In this embodiment, allocating multiple computing stages to multiple computing units within a distributed computing cluster includes: allocating multiple computing stages to multiple computing units within the distributed computing cluster. Each computing unit.

[0051] Furthermore, in this embodiment, for two consecutive adjacent mini-batches, scheduling the computational tasks of the final stage of the preceding mini-batch to be executed during the idle time slice of the preheating stage of the following mini-batch includes: for two consecutive adjacent mini-batches, scheduling the computational tasks of the final stage of the preceding mini-batch to be executed during the idle time slice of the preheating stage of the following mini-batch. Longer computational tasks are scheduled to the warm-up phase of the next mini-batch. Execution occurs during idle time slices within the specified duration, with the final phase having a time window length of [length missing]. The time window length of the preheating stage is , The duration of forward propagation computation for a single micro-batch. This represents the number of calculation units.

[0052] Figure 4 The diagram illustrates the pipeline for two adjacent mini-batches: mini-batch MP1 and mini-batch MP2. In this embodiment, the distributed computing cluster includes four computing units, namely GPU1, GPU2, GPU3, and GPU4. The model to be trained is divided into four computing stages along the network depth direction, and these four computing stages are assigned to the four computing units of the distributed computing cluster. The forward FLOPs of each computing stage are approximately equal, and the error threshold of FLOPs is... .like Figure 4 In this embodiment, each mini-batch of training data is divided into 4 micro-batches. The forward propagation computation time for a single micro-batch is... .

[0053] like Figure 2 ,exist Figure 2 In the embodiments described, the preheating stage for each mini-batch is the beginning. The duration of each mini-batch is determined by a specific phase, with the final phase at the end. The duration of the phase.

[0054] like Figure 4 In an alternative embodiment, for two consecutive mini-batches, mini-batch MP1 and mini-batch MP2, the computation tasks at the end of the previous mini-batch MP1 are scheduled in execution time to the idle time slice of the warm-up phase of the next mini-batch MP2. Specifically, the warm-up phase of the next mini-batch MP2 is the beginning of the next mini-batch MP2. The duration of the phase. In an alternative embodiment, the end of the original previous mini-batch MP1 is... The computation task for the duration phase (the end phase of the previous mini-batch MP1) is scheduled to the beginning of the next mini-batch MP2. The duration of the phase (the warm-up phase of the subsequent mini-batch MP2) allows the computational tasks of the final phase of the previous mini-batch MP1 to fill the original pipeline bubbles in the warm-up phase of the subsequent mini-batch MP2, thereby eliminating the original pipeline bubbles in the warm-up phase of the subsequent mini-batch. Originally, the final phase of the previous mini-batch MP1 and the warm-up phase of the subsequent mini-batch MP2 together had a total of... Duration, but after scheduling, only The corresponding calculation task can be completed in a short time.

[0055] In an alternative embodiment, two adjacent mini-batches are spliced ​​end-to-end. A mini batch, The total production line time for each mini-batch is ;in, To calculate the number of units, The number of micro-lots included in each mini-lot. This refers to the quantity of mini-batch production. Compared to techniques that do not perform end-to-end splicing, this alternative embodiment saves pipeline time, equivalent to accelerating training. Specifically, as... Figure 2 , Figure 4 In terms of the number of computing units The number of micro-lots in each mini-lot is 4. When it is 4, such as Figure 2 In the technical solution without end-to-end splicing, the production line time for each mini-batch is... ,like Figure 4 In the technical solution of splicing the beginning and end, Total production line time for each mini-batch The average production line time for each mini-batch is .like Figure 4 For example, when At that time, the average production line time for each mini-batch was The average production line time for each mini-batch is reduced by 10% compared to technologies that do not perform end-to-end splicing. For example, when At that time, the average production line time for each mini-batch was The average production line time for each mini-batch is reduced by 14% compared to technologies that do not perform end-to-end splicing. When the batch size is large, the average pipeline time for each mini-batch is close to 12 seconds, which theoretically reduces pipeline time by about 20%, i.e., speeds up training by about 25%.

[0056] Figure 5 This is a timing diagram of the pipeline between two adjacent mini-batches in another alternative embodiment of the model training method according to this application. Figure 5 In another alternative embodiment, some steps of the model training method are the same as... Figure 2 The steps involved in the embodiments are similar, and the following will mainly describe the differences between another alternative embodiment and the aforementioned embodiment.

[0057] In some embodiments, dividing the model to be trained into multiple computational stages along the network depth direction and distributing the multiple computational stages to multiple computational units of a distributed computing cluster includes: dividing the model to be trained into multiple computational stages along the network depth direction. Each computational stage, and will Each computation stage is assigned to a distributed computing cluster. The computational unit, in which the first The computing unit carries the first The first calculation stage and the first Each calculation stage, in which... It is a positive integer. From 1 to Integers.

[0058] In some embodiments, the model to be trained is divided along the network depth direction into Each calculation stage, The calculation stages are, for example, Stage-1, Stage-2, ..., Stage- Stage- Stage- Stage- .Will The computational phases are divided into two groups: Group 1 G1 and Group 2 G2. Group 1 G1 includes the preceding... The calculation stages are: Stage-1, Stage-2, ..., Stage- The second group, G2, includes the following... Each calculation stage: Stage- Stage- Stage- .

[0059] In Each computation stage is assigned to a distributed computing cluster. When there are multiple computing units, there are two mapping methods.

[0060] The first mapping method is sequential mapping, that is, the first... The computing unit carries the first The first calculation stage and the first Each computational stage, in terms of the number of computational units. The number of micro-lots in each mini-lot is 4. Taking 4 as an example, the model to be trained is divided into 8 computation stages along the network depth direction. When using the sequential mapping method, the first computing unit GPU1 carries the first computation stage Stage-1 and the fifth computation stage Stage-5; the second computing unit GPU2 carries the second computation stage Stage-2 and the sixth computation stage Stage-6; the third computing unit GPU3 carries the third computation stage Stage-3 and the seventh computation stage Stage-7; and the fourth computing unit GPU4 carries the fourth computation stage Stage-4 and the eighth computation stage Stage-8.

[0061] For the first mapping method, i.e., sequential mapping, when the four computing units complete the forward propagation computation of the eight computation stages, the data flow is: GPU1→GPU2→GPU3→GPU4→GPU1→GPU2→GPU3→GPU4. When the four computing units complete the activation value gradient calculation of the eight computation stages, the data flow is: GPU4→GPU3→GPU2→GPU1→GPU4→GPU3→GPU2→GPU1.

[0062] Preferably, the second mapping method is a symmetric mapping, that is, the first... The computing unit carries the first The first calculation stage and the first Each computational stage, in terms of the number of computational units. The number of micro-lots in each mini-lot is 4. Taking 4 as an example, the model to be trained is divided into 8 computation stages along the network depth direction. When using a symmetric mapping method, the first computing unit GPU1 carries the first computation stage Stage-1 and the eighth computation stage Stage-8; the second computing unit GPU2 carries the second computation stage Stage-2 and the seventh computation stage Stage-7; the third computing unit GPU3 carries the third computation stage Stage-3 and the sixth computation stage Stage-6; and the fourth computing unit GPU4 carries the fourth computation stage Stage-4 and the fifth computation stage Stage-5.

[0063] For the second mapping method, namely symmetric mapping, when the four computing units complete the forward propagation computation of the eight computation stages, the data flow is: GPU1→GPU2→GPU3→GPU4→GPU3→GPU2→GPU1. When the four computing units complete the activation value gradient calculation of the eight computation stages, the data flow is: GPU1→GPU2→GPU3→GPU4→GPU3→GPU2→GPU1.

[0064] like Figure 5 In another alternative embodiment, the second mapping method described above, namely symmetric mapping, is used. Figure 5 This illustrates a production line for a mini batch of MP1s.

[0065] In another alternative embodiment, the second mapping method described above, namely symmetric mapping, is used. This reduces the number of communications between multiple computing units when performing computations on each mini-batch. Specifically, after the forward computation reaches Stage-4 on the fourth computing unit GPU4, when executing Stage-5, the first mapping method (sequential mapping) requires passing the activation values ​​of Stage-4 to Stage-5 on the first computing unit GPU1. However, in the second mapping method (symmetric mapping), Stage-5 continues to be executed on the fourth computing unit GPU4. Similarly, for calculating activation value gradients, the second mapping method (symmetric mapping) also reduces the amount of communication between computing units compared to the first mapping method (sequential mapping).

[0066] Furthermore, in some embodiments, the alternative embodiment described above may also incorporate the following technical solution: for two consecutive adjacent mini-batches, the computational tasks of the final stage of the preceding mini-batch are scheduled to be executed during the idle time slice of the warm-up stage of the following mini-batch. For example, scheduling the computational tasks of the final stage of the preceding mini-batch to be executed during the idle time slice of the warm-up stage of the following mini-batch for two consecutive adjacent mini-batches includes: for two consecutive adjacent mini-batches, scheduling the computational tasks of the final stage of the preceding mini-batch to be executed during the idle time slice of the warm-up stage of the following mini-batch ... scheduling the computational tasks of the final stage of the preceding mini-batch to be executed during the idle time slice of the warm-up stage of the following mini-batch. Longer computational tasks are scheduled to the warm-up phase of the next mini-batch. Execution occurs during idle time slices within the specified duration, with the final phase having a time window length of [length missing]. The time window length of the preheating stage is , The duration of forward propagation computation for a single micro-batch. This represents the number of calculation units.

[0067] This application also provides a model training device. Figure 6 This is a schematic diagram of one embodiment of the model training apparatus according to this application. The model training apparatus includes an allocation module 110, a training data processing module 120, and a training execution module 130.

[0068] The allocation module 110 is used to divide the model to be trained into multiple computational stages along the network depth direction, and allocate the multiple computational stages to multiple computational units of the distributed computing cluster.

[0069] The training data processing module 120 is used to divide each mini-batch of training data into multiple micro-batches.

[0070] The training execution module 130 is used to perform forward propagation computation and backward propagation computation sequentially on micro-batches based on multiple computation stages and micro-batches.

[0071] Figure 7 This is a schematic diagram of the structure of the training execution module in one embodiment of the model training apparatus according to this application. The training execution module 130 includes a forward computation submodule 131 and a backward computation submodule 132.

[0072] The forward computation submodule 131 is used to perform forward propagation computation, so that multiple micro-batches of each mini-batch pass through multiple computation units in the forward direction of the network depth.

[0073] The backpropagation submodule 132 is used to perform backpropagation calculation. The backpropagation submodule 132 includes an activation gradient calculation unit 1321 and a weight gradient calculation unit 1322.

[0074] The activation value gradient calculation unit 1321 is used to perform activation value gradient calculation, so that multiple micro-batches of each mini-batch pass through multiple calculation units in reverse order of network depth, and receive activation value gradients from the next adjacent calculation stage in each calculation stage, and calculate the activation value gradient of the current calculation stage.

[0075] The weight gradient calculation unit 1322 is used to perform weight gradient calculation, wherein, in each calculation stage, the weight gradient of the current calculation stage is calculated based on the intermediate results of the local gradient obtained by the activation value gradient calculation unit and the corresponding forward propagation activation values.

[0076] In this embodiment, the weight gradient calculation is initiated without waiting for all activation value gradient calculations to be completed, based on the already obtained corresponding activation value gradient results.

[0077] According to the model training apparatus of this application embodiment, the training execution module 130 includes a forward computation submodule 131 and a backward computation submodule 132. The forward computation submodule 131 performs forward propagation computation, causing multiple micro-batches of each mini-batch to sequentially pass through multiple computation units in the forward direction of the network depth. The backward computation submodule 132 performs backward propagation computation, and includes an activation value gradient calculation unit 1321 and a weight gradient calculation unit 1322. The activation value gradient calculation unit 1321 performs activation value gradient calculation, causing multiple micro-batches of each mini-batch to sequentially pass through multiple computation units in the backward direction of the network depth, and receives activation value gradients from the adjacent next computation stage at each computation stage, and calculates the activation value gradient of the current computation stage. The weight gradient calculation unit 1322 performs weight gradient calculation, wherein, at each computation stage, the weight gradient of the current computation stage is calculated based on the intermediate local gradient results obtained by the activation value gradient calculation unit and the corresponding forward propagation activation values. Weight gradient calculation is initiated after obtaining the corresponding activation value gradient results, without waiting for all activation value gradient calculations to complete. Therefore, when performing calculations on each mini-batch, the traditional backpropagation calculation is split into two logically and temporally independent sub-steps: activation value gradient calculation and weight gradient calculation. Compared to the traditional approach of encapsulating activation value gradient calculation and weight gradient calculation together, this embodiment reduces the waiting time between activation value gradient calculation and weight gradient calculation, thereby reducing the number of pipeline bubbles in the pipeline parallel training process and improving the utilization of distributed computing resources. On the other hand, when the weight gradient calculation unit 1322 performs weight gradient calculation, it does so based on the intermediate results of local gradients during the activation value gradient calculation process and the local forward propagation activation values, without the need for cross-computation unit communication, thus reducing communication overhead. Furthermore, for each calculation stage, the cache occupancy of the corresponding computing unit can be released immediately after obtaining the weight gradient, such as releasing the GPU's video memory occupancy, reducing peak cache requirements.

[0078] This application also provides an electronic device. Figure 8This is a schematic diagram of the hardware structure of an embodiment of the electronic device according to this application. The electronic device includes a memory 910 and a processor 920. The memory 910 and the processor 920 are communicatively connected. The memory 910 stores instructions, and the processor 920 calls the instructions in the memory 910 to cause the electronic device to execute the model training method according to any of the foregoing embodiments of this application.

[0079] The model training method includes: dividing the model to be trained into multiple computational stages along the network depth direction, and allocating these stages to multiple computational units in a distributed computing cluster; dividing each mini-batch of training data into multiple micro-batches; and sequentially performing forward propagation and backward propagation computations on the micro-batches based on the multiple computational stages and micro-batches; in the forward propagation computation step, the multiple micro-batches of each mini-batch pass through multiple computational units sequentially in the forward direction of the network depth direction; the backward propagation computation step includes activation value gradient calculation and weight gradient calculation; in the activation value gradient calculation step, the multiple micro-batches of each mini-batch pass through multiple computational units sequentially in the backward direction of the network depth direction, receiving activation value gradients from the adjacent next computational stage at each computational stage, and calculating the activation value gradient of the current computational stage; in the weight gradient calculation step, at each computational stage, the weight gradient of the current computational stage is calculated based on the intermediate local gradient results obtained from the activation value gradient calculation step and the corresponding forward propagation activation values; wherein, the weight gradient calculation is initiated based on the obtained corresponding activation value gradient results without waiting for all activation value gradient calculations to be completed.

[0080] Specifically, the processor 920 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0081] Memory 910 may include a large-capacity memory for data or instructions. For example, and not limitingly, memory 910 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 910 may include removable or non-removable (or fixed) media. Where appropriate, memory 910 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 910 may be the non-volatile memory described above. In a particular embodiment, memory 910 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0082] In one example, the electronic device may also include a communication interface 930 and a bus 940. The processor 920, memory 910, and communication interface 930 are connected via the bus 940 and communicate with each other.

[0083] The communication interface 930 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0084] Bus 940 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 940 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0085] Furthermore, in conjunction with the model training methods described in the above embodiments, this application can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores instructions that, when executed by a processor, implement the model training method of any of the foregoing embodiments of this application.

[0086] The model training method includes: dividing the model to be trained into multiple computational stages along the network depth direction, and allocating these stages to multiple computational units in a distributed computing cluster; dividing each mini-batch of training data into multiple micro-batches; and sequentially performing forward propagation and backward propagation computations on the micro-batches based on the multiple computational stages and micro-batches; in the forward propagation computation step, the multiple micro-batches of each mini-batch pass through multiple computational units sequentially in the forward direction of the network depth direction; the backward propagation computation step includes activation value gradient calculation and weight gradient calculation; in the activation value gradient calculation step, the multiple micro-batches of each mini-batch pass through multiple computational units sequentially in the backward direction of the network depth direction, receiving activation value gradients from the adjacent next computational stage at each computational stage, and calculating the activation value gradient of the current computational stage; in the weight gradient calculation step, at each computational stage, the weight gradient of the current computational stage is calculated based on the intermediate local gradient results obtained from the activation value gradient calculation step and the corresponding forward propagation activation values; wherein, the weight gradient calculation is initiated based on the obtained corresponding activation value gradient results without waiting for all activation value gradient calculations to be completed.

[0087] This application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0088] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0089] It should be noted that the technical solutions or features described in the above embodiments can be combined or supplemented with each other without conflict. The scope of protection of this application is not limited to the precise structures described in the above embodiments and shown in the accompanying drawings; all modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A model training method, characterized in that, include: The model to be trained is divided into multiple computational stages along the network depth direction, and the multiple computational stages are distributed to multiple computing units of a distributed computing cluster. Divide each mini-batch of training data into multiple micro-batches; as well as Based on the multiple computational stages and the micro-batch, forward propagation computation and backward propagation computation are sequentially performed on the micro-batch. In the forward propagation calculation step, the plurality of micro-batches of each mini-batch are sequentially passed through the plurality of calculation units in the forward direction of the network depth direction; The backpropagation calculation steps include activation gradient calculation and weight gradient calculation; In the step of calculating the activation value gradient, the plurality of micro-batches of each mini-batch are sequentially passed through the plurality of computing units in the reverse direction of the network depth. At each computing stage, the activation value gradient from the next adjacent computing stage is received, and the activation value gradient of the current computing stage is calculated. In the weight gradient calculation step, at each calculation stage, the weight gradient of the current calculation stage is calculated based on the intermediate local gradient results obtained from the activation value gradient calculation step and the corresponding forward propagation activation values. The weight gradient calculation is initiated after the corresponding activation value gradient results have been obtained, without waiting for the completion of the gradient calculation of all activation values.

2. The model training method according to claim 1, characterized in that, The forward propagation calculation includes: In each of the computation stages, the activation value of the adjacent previous computation stage is received; Based on the received activation values ​​of the adjacent previous computation stage and the weight matrix of the current computation stage, calculate the linear output of the current computation stage. The activation value for the current calculation stage is generated based on the linear output and activation function. Output the activation value of the current calculation stage to the next adjacent calculation stage.

3. The model training method according to claim 2, characterized in that, The activation value gradient calculation includes: Receive the activation value gradient from the next adjacent computation stage, which corresponds to the activation value output by the current computation stage; Based on the activation value gradient from the next adjacent computation stage, the gradient calculation rule of the activation function, and the linear output obtained by the current computation stage in the forward propagation calculation, the local gradient intermediate result of the current computation stage is generated. Based on the intermediate results of the local gradient and the weight matrix of the current calculation stage, the activation value gradient corresponding to the input activation value of the current calculation stage is calculated. The activation value gradient calculated in the current calculation stage is output to the adjacent previous calculation stage.

4. The model training method according to claim 3, characterized in that, The weight gradient calculation includes: In each computation stage, the weight gradient of the current computation stage is calculated based on the intermediate results of the local gradient and the activation values ​​of the adjacent previous computation stage.

5. The model training method according to claim 1, characterized in that, Also includes: For two consecutive adjacent mini-batches, the computational tasks at the end of the previous mini-batch are scheduled to be executed during the idle time slice of the warm-up phase of the next mini-batch.

6. The model training method according to claim 5, characterized in that, The allocation of the multiple computing stages to multiple computing units in a distributed computing cluster includes: Distribute the multiple computing stages to a distributed computing cluster. One computing unit; The step of scheduling the computational tasks at the end of the previous mini-batch to be executed during the idle time slice of the warm-up phase of the subsequent mini-batch for two consecutive adjacent mini-batches includes: For two consecutive adjacent mini-batches, the end phase of the previous mini-batch will be... The computational tasks with longer durations are scheduled to the warm-up phase of the mini-batch described later. Execution is performed during idle time slices within the specified duration, wherein the time window length of the final stage is [length missing]. The time window length of the preheating stage is , The duration of forward propagation computation for a single micro-batch. The number of computing units.

7. The model training method according to claim 1, characterized in that, The step of dividing the model to be trained into multiple computational stages along the network depth direction and distributing the multiple computational stages to multiple computing units of a distributed computing cluster includes: The model to be trained is divided along the network depth direction. Each calculation stage, and the... Each computation stage is assigned to a distributed computing cluster. The computational unit, wherein the first... The computing unit carries the first Each of the aforementioned computational stages, From 1 to Integers.

8. The model training method according to claim 1, characterized in that, The step of dividing the model to be trained into multiple computational stages along the network depth direction and distributing the multiple computational stages to multiple computing units of a distributed computing cluster includes: The model to be trained is divided along the network depth direction. Each calculation stage, and the... Each computation stage is assigned to a distributed computing cluster. The computational unit, wherein the first... The computing unit carries the first The computational stage and the first Each of the aforementioned computational stages, From 1 to Integers.

9. A model training device, characterized in that, include: The allocation module is used to divide the model to be trained into multiple computing stages along the network depth direction, and allocate the multiple computing stages to multiple computing units of the distributed computing cluster. The training data processing module is used to divide each mini-batch of training data into multiple micro-batches; as well as The training execution module is used to sequentially perform forward propagation and backward propagation calculations on the micro-batches based on the multiple computational stages and the micro-batches. The training execution module includes: The forward computation submodule is used to perform forward propagation computation, so that multiple micro-batches of each mini-batch pass through the multiple computation units in the forward direction of the network depth direction. The back propagation submodule is used to perform backpropagation calculations. The back propagation submodule includes an activation gradient calculation unit and a weight gradient calculation unit. The activation value gradient calculation unit is used to perform activation value gradient calculation, so that multiple micro-batches of each mini-batch pass through the multiple calculation units in the reverse direction of the network depth, and receive activation value gradients from the next adjacent calculation stage in each calculation stage, and calculate the activation value gradient of the current calculation stage. The weight gradient calculation unit is used to perform weight gradient calculation, wherein, in each calculation stage, the weight gradient of the current calculation stage is calculated based on the intermediate local gradient results obtained by the activation value gradient calculation unit and the corresponding forward propagation activation values. The weight gradient calculation is initiated after the corresponding activation value gradient results have been obtained, without waiting for the completion of the gradient calculation of all activation values.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being communicatively connected to the processor, the memory storing instructions, and the processor calling the instructions in the memory to cause the electronic device to execute the model training method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the model training method according to any one of claims 1 to 8.

Images