Model training acceleration method and device, electronic equipment, medium and product

Abstract

The invention discloses a model training acceleration method and device, electronic equipment, a medium and a product. The method comprises the following steps: executing initialization of a model training process including pipeline parallelism; in the first stage of each assembly line, before the forward propagation calculation of the current microbatch is executed, the forward propagation calculation result of the next microbatch is received in a non-blocking mode, and after the forward propagation calculation of the current microbatch is completed, the forward propagation calculation result of the current microbatch is sent in a non-blocking mode. And overlapping execution of calculation and communication is realized. According to the model training acceleration method provided by the embodiment of the invention, the non-blocking communication mechanism is introduced in the pipeline parallel training process, so that the calculation and communication overlapping of the calculation nodes can be realized, the network bandwidth and delay requirements on the calculation nodes are greatly reduced, the waiting time of node calculation is reduced, and the calculation efficiency is improved. Therefore, the model training efficiency and the computing power utilization rate of the computing nodes are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, medium, and product for accelerating model training. Background Technology

[0002] By employing pipelined parallelism, a small batch of training samples can be further divided into multiple micro batches, and the model parameters of the model to be trained can be distributed across multiple devices. The model training process is carried out in micro batches as the basic unit, forming a pipeline that includes computation and communication on multiple computing nodes. This can reduce the memory burden on a single computing node and improve the efficiency of model training.

[0003] Existing pipelined parallel acceleration methods employ a serial execution approach for computation and communication. This serial execution approach requires significant network bandwidth and very low network latency between computing nodes. When using this serial execution approach in a low-bandwidth, high-latency network environment, the waiting time for node computation increases (because node computation needs to be performed after node communication is completed), thereby significantly slowing down the entire model training process and substantially reducing the computing power utilization of computing nodes. Summary of the Invention

[0004] This invention provides a model training acceleration method, device, electronic device, medium, and product to solve the problems of low model training efficiency and low node computing power utilization in existing solutions.

[0005] According to one aspect of the present invention, a method for accelerating model training is provided, executed by a computing node, the method comprising:

[0006] Perform initialization of the model training process, which includes pipelined parallelism;

[0007] In the first stage of each pipeline, before executing the forward propagation computation of the current micro-batch, the forward propagation computation result of the next micro-batch is received non-blockingly, and after completing the forward propagation computation of the current micro-batch, the forward propagation computation result of the current micro-batch is sent non-blockingly, so as to achieve overlapping execution of computation and communication.

[0008] According to another aspect of the present invention, a model training acceleration device is provided, configured in a computing node, the device comprising:

[0009] The model training initialization module is used to perform the initialization of the model training process, which includes pipelined parallelism.

[0010] The computation and communication overlapped execution module is used to receive the forward propagation calculation results of the next micro-batch in non-blocking before executing the forward propagation calculation of the current micro-batch in the first stage of each pipeline, and to send the forward propagation calculation results of the current micro-batch in non-blocking after completing the forward propagation calculation of the current micro-batch, so as to achieve overlapped execution of computation and communication.

[0011] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0012] At least one processor; and

[0013] A memory communicatively connected to the at least one processor; wherein,

[0014] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the model training acceleration method according to any embodiment of the present invention.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the model training acceleration method according to any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the model training acceleration method described in any embodiment of the present invention.

[0017] The model training acceleration method provided in this invention initializes a pipelined parallel model training process. Within the first stage of each pipeline, before executing the forward propagation computation of the current micro-batch, the forward propagation computation result of the next micro-batch is received non-blockingly, and after completing the forward propagation computation of the current micro-batch, the forward propagation computation result of the current micro-batch is sent non-blockingly, thus achieving overlapping execution of computation and communication. This technical solution, by introducing a non-blocking communication mechanism during pipelined parallel training, enables overlapping of computation and communication between computing nodes, significantly reducing the network bandwidth and latency requirements of computing nodes, reducing node computation waiting time, and thereby improving model training efficiency and the utilization rate of computing node computing power.

[0018] The model training acceleration device, electronic device, computer-readable storage medium, and computer program product provided in this embodiment of the invention also possess the above-mentioned technical effects.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of a serial computing and communication pipeline provided according to Embodiment 1 of the present invention;

[0022] Figure 2 This is a schematic diagram of a pipeline that overlaps computation and communication according to Embodiment 1 of the present invention;

[0023] Figure 3 This is a flowchart of a model training acceleration method provided in Embodiment 1 of the present invention;

[0024] Figure 4 This is a flowchart of a first stage provided according to Embodiment 1 of the present invention;

[0025] Figure 5 This is a flowchart of a second stage provided according to Embodiment 1 of the present invention;

[0026] Figure 6 This is a flowchart of a third stage provided according to Embodiment 1 of the present invention;

[0027] Figure 7 This is a flowchart of a model training acceleration method provided in Embodiment 2 of the present invention;

[0028] Figure 8 This is a flowchart of a first stage provided according to Embodiment 2 of the present invention;

[0029] Figure 9 This is a flowchart of a second stage provided according to Embodiment 2 of the present invention;

[0030] Figure 10 This is a flowchart of a third stage provided according to Embodiment 2 of the present invention;

[0031] Figure 11 This is a timing diagram of a model training acceleration method provided according to Embodiment 2 of the present invention;

[0032] Figure 12This is a schematic diagram of the structure of a model training acceleration device according to Embodiment 3 of the present invention;

[0033] Figure 13 This is a schematic diagram of the structure of an electronic device that implements the model training acceleration method of this invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] Example 1

[0037] like Figure 1 As shown, existing pipelined parallel acceleration methods employ a serial execution approach for computation and communication. This serial execution approach requires significant network bandwidth and very low network latency between computation nodes (workers). When using this serial execution approach in a low-bandwidth, high-latency network environment, the waiting time for node computation increases (because node computation needs to be performed after node communication is completed), thereby significantly slowing down the entire model training process and substantially reducing the computational efficiency of the computation nodes.

[0038] To solve the above problems, such as Figure 2 As shown in the figure, this embodiment of the invention proposes a model training acceleration method. By introducing a non-blocking communication mechanism, the computation and communication of computing nodes are overlapped, which greatly reduces the network bandwidth and latency requirements of computing nodes, reduces the waiting time of node computation, and thus improves the efficiency of model training and the computing power utilization of computing nodes.

[0039] Figure 3This is a flowchart of a model training acceleration method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where model training is accelerated by introducing a non-blocking communication mechanism to overlap computation and communication in low-bandwidth and high-latency network environments. The method can be executed by a model training acceleration device, which can be implemented in hardware and / or software. This device can be configured in electronic devices, such as, but not limited to, computers and servers. Figure 3 As shown in the figure, the model training acceleration method provided in this embodiment can be executed by a computing node. The method specifically includes the following steps:

[0040] S110. Execute the initialization of the model training process, which includes pipelined parallelism.

[0041] Here, a computing node (worker) can refer to an independent hardware or software entity that undertakes specific computing tasks in the pipelined parallel training of the model. For example, a computing node may include a graphics processing unit (GPU) server, a tensor processing unit (TPU) server, etc.

[0042] In this embodiment of the invention, before executing the pipeline parallel process of the model to be trained, an initialization configuration operation needs to be performed first. This may include, but is not limited to, local model loading and parameter initialization, division of the number of micro-batches in each stage of the pipeline, determination of the order of computing nodes (i.e., the position of each computing node in the pipeline), configuration of communication links between nodes, memory management initialization, and computing resource initialization.

[0043] It is important to understand that the initialization process during model training is a routine operation in existing model training and is not the focus of this invention. For specific implementation details, please refer to existing technologies; they will not be elaborated upon here. The model to be trained generally refers to a large-scale machine learning model (e.g., with hundreds of millions of parameters), such as a large language model based on Transformer (e.g., GPT, LlaMA). The training data used can be text data, such as a sentence, an article, or a question.

[0044] S120. In the first stage of each pipeline, before executing the forward propagation calculation of the current micro-batch, the forward propagation calculation result of the next micro-batch is received non-blockingly, and after completing the forward propagation calculation of the current micro-batch, the forward propagation calculation result of the current micro-batch is sent non-blockingly, so as to achieve overlapping execution of calculation and communication.

[0045] The first stage can refer to the initial stage of pipelined parallel training, also known as the warm-up stage. In this stage, forward propagation computation is performed as much as possible by increasing the number of micro-batches to gradually fill the pipeline and start non-blocking communication for forward propagation computation. Backward propagation computation has not yet started.

[0046] Forward propagation computation refers to the process in which input data sequentially passes through each layer of the model during forward propagation, ultimately generating intermediate activation values ​​or the final output result. In pipelined parallel acceleration, this process is executed on computing nodes such as GPUs, using micro-batches as the basic unit. The computation results (intermediate activation values ​​or the final output result) are sent to the next-level computing node via non-blocking communication for subsequent processing.

[0047] Backpropagation computation refers to the process of calculating the gradient of the loss function with respect to the model parameters in micro-batches during the backpropagation of the model, using the chain rule. These gradients can be used to update the model parameters. In pipelined parallel acceleration, the backpropagation computation results are sent to the next-level computing node via non-blocking communication. Its computation process can overlap with GPU communication to improve the training efficiency of large models and the utilization of GPU computing power.

[0048] Non-blocking communication, as an asynchronous communication technology, allows computing nodes to immediately return to perform other computational operations without waiting for communication to complete, thereby achieving overlapping execution of computation and communication. Non-blocking communication can be triggered before and / or after performing forward / backward propagation computation operations. For example, before performing the forward propagation computation of the current micro-batch, a non-blocking reception operation of the forward propagation computation result of the next micro-batch can be initiated; after the forward propagation computation of the current micro-batch is completed, a non-blocking transmission operation of the forward propagation computation result of the current micro-batch can be initiated. In one embodiment, the computing node can implement related non-blocking communication operations by calling corresponding preset asynchronous communication interfaces, such as non-blocking reception of forward / backward propagation computation results, non-blocking transmission of forward / backward propagation computation results, etc. The preset asynchronous communication interfaces may include, but are not limited to, the NVIDIA Collective Communications Library (NCCL) interface, the Message Passing Interface (MPI), etc.

[0049] In this embodiment of the invention, in the first stage of each pipeline, before each execution of the forward propagation calculation of the current micro-batch, the forward propagation calculation result of the next micro-batch can be received non-blockingly; after the forward propagation calculation of the current micro-batch is completed, the forward propagation calculation result of the current micro-batch can be sent non-blockingly, so that other calculation operations can be executed immediately, realizing the overlap of calculation and communication.

[0050] The model training acceleration method provided in this invention initializes a pipelined parallel model training process. Within the first stage of each pipeline, before executing the forward propagation computation of the current micro-batch, the forward propagation computation result of the next micro-batch is received non-blockingly, and after completing the forward propagation computation of the current micro-batch, the forward propagation computation result of the current micro-batch is sent non-blockingly, thus achieving overlapping execution of computation and communication. This technical solution, by introducing a non-blocking communication mechanism during pipelined parallel training, enables overlapping of computation and communication between computing nodes, significantly reducing the network bandwidth and latency requirements of computing nodes, reducing node computation waiting time, and thereby improving model training efficiency and the utilization rate of computing node computing power.

[0051] Furthermore, based on the above embodiments of the invention, such as Figure 4 As shown, the first stage of the production line specifically includes the following steps:

[0052] S210. Obtain the number of micro-batches corresponding to the first, second and third stages of the pipeline, respectively.

[0053] S220. Initialize the forward micro-batch number N and the backward micro-batch number M;

[0054] S230: Receive the target forward propagation calculation result of the Nth micro-batch from the previous computing node;

[0055] S240. If the current Nth microbatch is not the last microbatch in the current training iteration, then start the non-blocking reception operation of the forward propagation calculation result of the N+1th microbatch.

[0056] S250. Execute the forward propagation calculation for the Nth micro-batch based on the forward propagation calculation results of the target;

[0057] S260. If the Nth micro-batch is not the last micro-batch in the first stage, then start the non-blocking transmission operation of the forward propagation calculation result of the Nth micro-batch, and after waiting for the non-blocking reception operation of the forward propagation calculation result of the N+1th micro-batch to be completed, increment the forward micro-batch number N, and return to the step of starting the non-blocking reception operation of the forward propagation calculation result of the N+1th micro-batch if the current Nth micro-batch is not the last micro-batch in the current training iteration.

[0058] S270. If the Nth micro-batch is the last micro-batch in the first stage, then initiate the non-blocking transmission operation of the forward propagation calculation result of the Nth micro-batch and initiate the non-blocking reception operation of the backward propagation calculation result of the Mth micro-batch. If the number of micro-batches in the second stage is not 0, then proceed to the second stage, or if the number of micro-batches in the second stage is 0, then proceed to the third stage.

[0059] The second stage can refer to the core stage of pipelined parallel training. The second stage can also be called the stabilization stage. In this stage, the pipeline runs at full speed, and forward propagation calculation and backward propagation calculation are executed in parallel to maximize the training efficiency of large models and the utilization rate of node computing power.

[0060] The third stage can refer to the final stage of pipelined parallel training. The third stage can also be called the cooling stage. In this stage, the computing nodes mainly process the remaining backpropagation computing tasks and clear the unfinished computing tasks in the pipeline.

[0061] Understandably, not all computing nodes participating in pipelined parallel training include all three training phases (Phase 1, Phase 2, and Phase 3). Depending on the initial configuration of each computing node, some nodes may only have Phase 1 and Phase 2. In this case, the backpropagation computation of all micro-batches of the computing node will be completed within Phase 2. Other computing nodes may only have Phase 1 and Phase 3. In this case, the computing node will directly enter Phase 3 after completing Phase 1 and complete the backpropagation computation of the corresponding micro-batches. Typically, each computing node includes Phase 1 and at least one of Phase 2 and Phase 3.

[0062] The forward micro-batch number N and the backward micro-batch number M represent the micro-batch numbers for forward and backward propagation, respectively. They can be used to track the progress of micro-batch processing at different training stages, ensure the correctness of data dependencies, and avoid pipeline conflicts. For example, ① forward propagation dependency: This computing node can only execute the forward propagation calculation of the Nth micro-batch after receiving the forward propagation result of the Nth micro-batch from the previous computing node; ② backward propagation dependency: This computing node can only execute the backward propagation calculation of the Mth micro-batch after receiving the backward propagation result of the Mth micro-batch from the next computing node.

[0063] In this embodiment of the invention, the first stage of the pipeline may include the following process:

[0064] (1) The current computing node can obtain the number of micro-batches configured for the first, second, and third stages of the pipeline during the model training initialization phase. In one embodiment, the number of micro-batches corresponding to each training stage can be determined according to a pre-configured micro-batchase allocation strategy. For example, the number of micro-batches for each training stage can be allocated based on pipeline depth (i.e., the number of computing nodes) or a fixed ratio strategy, or the number of micro-batches for each target training stage can be dynamically allocated based on indicators such as node load and communication latency. It is worth understanding that in the first stage, the corresponding number of micro-batches can be increased as much as possible, that is, as much forward propagation computation as possible, to ensure that the pipeline is filled quickly and to reduce bubbles in subsequent training stages.

[0065] It is important to understand that the number of micro-batches for each training stage varies depending on the location of each computing node in the pipeline.

[0066] In one specific embodiment, the number of micro-batches corresponding to each computing node in the pipeline for the first, second, and third stages can be determined in the following manner:

[0067] Assuming the pipeline depth is D and the total number of micro-batches is B, then for a computation node at position r (0≤r≤D-1) in the pipeline, the number of micro-batches in the first, second, and third stages are 3(Dr-1), B-3(Dr-1), and 3(Dr-1), respectively. That is, the number of micro-batches in the second stage is equal to the total number of micro-batches B minus the number of micro-batches in the first stage, and the number of micro-batches in the third stage is equal to the number of micro-batches in the first stage.

[0068] It is understood that the above method for determining the number of micro-batches in each training stage of the pipeline is only an example. In practical applications, the number of micro-batches in each training stage can also be determined in other ways, and this embodiment does not limit this.

[0069] (2) The pre-configured forward micro-batch number N and backward micro-batch number M can be initialized by setting N=1 and M=1.

[0070] (3) Receive the target forward propagation calculation results of the Nth micro-batch sent by the previous computing node.

[0071] (4) Determine whether the current Nth micro-batch is the last micro-batch in the current training iteration. If yes, proceed to the next step; otherwise, start the non-blocking reception operation of the forward propagation calculation result of the N+1th micro-batch and then proceed to the next step.

[0072] (5) Perform the forward propagation calculation of the Nth micro-batch based on the local model and the forward propagation calculation results of the target to obtain the corresponding forward propagation calculation results.

[0073] (6) Determine whether the current Nth micro-batch is the last micro-batch of the first stage.

[0074] (7) If so, start the non-blocking transmission operation of the forward propagation calculation result of the Nth micro-batch and start the non-blocking reception operation of the backward propagation calculation result of the Mth micro-batch; at the same time, determine whether the number of micro-batch corresponding to the second stage is 0. If the number of micro-batch in the second stage is not 0, the current computing node will switch from the first stage to the second stage; or, if the number of micro-batch in the second stage is 0, the current computing stage will switch from the first stage to the third stage.

[0075] (8) If not, start the non-blocking transmission operation of the forward propagation calculation result of the Nth micro-batch, and wait for the non-blocking reception operation of the forward propagation calculation result of the N+1th micro-batch to be completed. Then increment the forward micro-batch number N, i.e., set N = N+1, and return to execute the above step (4).

[0076] Furthermore, based on the above embodiments of the invention, such as Figure 5 As shown, the production line also includes a second stage, which specifically includes the following steps:

[0077] S310. Obtain the number of micro-batches corresponding to the second and third stages in the pipeline, and obtain the forward micro-batch number N and the backward micro-batch number M after the first stage.

[0078] S320, Wait for the non-blocking reception operation of the target forward propagation calculation result of the Nth micro-batch to complete;

[0079] S330. Based on the current position of the Nth micro-batch in the second stage, initiate the corresponding non-blocking receive operation or non-blocking transmit operation.

[0080] S340. Execute the forward propagation calculation for the Nth micro-batch based on the forward propagation calculation results of the target;

[0081] S350. After the non-blocking reception operation of the backpropagation calculation result of the target of the Mth micro-batch is completed, the non-blocking transmission operation of the forward propagation calculation result of the Nth micro-batch is started, and the non-blocking reception operation of the backpropagation calculation result of the M+1th micro-batch is started.

[0082] S360. Perform the backward propagation calculation of the Mth micro-batch based on the backpropagation calculation result of the target, and increment the forward micro-batch number N and the backward micro-batch number M after the calculation is completed.

[0083] S370. If the Nth micro-batch is not the last micro-batch of the second stage, return to the step of waiting for the non-blocking receive operation of the target forward propagation calculation result of the Nth micro-batch to complete.

[0084] S380. If the Nth micro-batch is the last micro-batch of the second stage and the number of micro-batches in the third stage is not 0, then start the non-blocking transmission operation of the backpropagation calculation result of the M-1th micro-batch and proceed to the third stage.

[0085] S390. If the Nth micro-batch is the last micro-batch of the second stage and the number of micro-batches in the third stage is 0, then send the backpropagation calculation result of the M-1th micro-batch to the next higher-level computing node and stop the pipeline operation.

[0086] In this embodiment of the invention, the pipelined parallel training may further include a second stage, which may include the following process:

[0087] (1) Obtain the number of micro-batches corresponding to the second and third stages in the pipeline, and at the same time obtain the latest forward micro-batch number N and backward micro-batch number M after the first stage.

[0088] (2) Wait for the non-blocking reception operation of the target forward propagation calculation result of the Nth micro-batch to complete.

[0089] (3) Based on the position of the current Nth micro-batch in the second stage, initiate the corresponding non-blocking receive operation or non-blocking transmit operation. Specifically, if the Nth micro-batch is the first micro-batch in the second stage, initiate the non-blocking receive operation of the forward propagation calculation result of the N+1th micro-batch; if the Nth micro-batch is neither the first nor the last micro-batch in the second stage, initiate the non-blocking transmit operation of the backward propagation calculation result of the M-1th micro-batch, and initiate the non-blocking receive operation of the forward propagation calculation result of the N+1th micro-batch; if the Nth micro-batch is the last micro-batch in the second stage, initiate the non-blocking transmit operation of the backward propagation calculation result of the M-1th micro-batch.

[0090] (4) Perform the forward propagation calculation of the Nth micro-batch based on the local model and the forward propagation calculation results of the target to obtain the corresponding forward propagation calculation results.

[0091] (5) Wait for the non-blocking reception operation of the backpropagation calculation result of the target in the Mth micro-batch to complete.

[0092] (6) Initiate the non-blocking transmission operation of the forward propagation calculation results of the Nth micro-batch, and initiate the non-blocking reception operation of the backward propagation calculation results of the M+1th micro-batch.

[0093] (7) Perform the backpropagation calculation of the Mth micro-batch based on the local model and the backpropagation calculation results of the target to obtain the corresponding backpropagation calculation results.

[0094] (8) Increment the forward micro-batch number N and the backward micro-batch number M, i.e., let N = N + 1 and M = M + 1.

[0095] (9) If the current Nth micro-batch is not the last micro-batch in the stable phase, then return to the above step (2).

[0096] (10) If the current Nth micro-batch is the last micro-batch in the stable phase and the number of micro-batches in the cooling phase is not 0, then start the non-blocking transmission operation of the backpropagation calculation result of the M-1th micro-batch and switch from the second phase to the third phase.

[0097] (11) If the current Nth micro-batch is the last micro-batch in the stable phase and the number of micro-batches in the cooling phase is 0, then the backpropagation calculation result of the M-1th micro-batch is sent directly to the next higher-level computing node, and the pipeline operation is stopped.

[0098] Furthermore, based on the above embodiments of the invention, such as Figure 6 As shown, the production line also includes a third stage, which specifically includes the following steps:

[0099] S410. Obtain the forward micro-batch number N and the backward micro-batch number M after the first stage, or the forward micro-batch number N and the backward micro-batch number M after the first stage and the second stage.

[0100] S420, Wait for the non-blocking reception operation of the target backpropagation calculation result of the Mth micro-batch to complete;

[0101] S430. If the current Mth micro-batch is not the last micro-batch of the third stage, then start the non-blocking reception operation of the backpropagation calculation result of the M+1th micro-batch, and perform the backpropagation calculation of the Mth micro-batch according to the target backpropagation calculation result. Then start the non-blocking transmission operation of the backpropagation calculation result of the Mth micro-batch, increment the backpropagation micro-batch number M, and return to the step of waiting for the completion of the non-blocking reception operation of the target backpropagation calculation result of the Mth micro-batch.

[0102] S440. If the current Mth micro-batch is the last micro-batch of the third stage, then perform the backpropagation calculation of the Mth micro-batch according to the target backpropagation calculation result, send the backpropagation calculation result of the Mth micro-batch to the next higher level computing node, and stop pipeline operation.

[0103] In this embodiment of the invention, the pipelined parallel training may further include a third stage, which may include the following process:

[0104] (1) Determine whether the number of micro-batches corresponding to the second stage of the current computing node is zero. If yes (i.e. there is no second stage), obtain the latest forward micro-batch number N and backward micro-batch number M after the first stage. If no, obtain the latest forward micro-batch number N and backward micro-batch number M after the first and second stages.

[0105] (2) Wait for the non-blocking reception operation of the backpropagation calculation result of the target in the Mth micro-batch to complete.

[0106] (3) Determine whether the current Mth micro-batch is the last micro-batch of the third stage. If yes, proceed to the next step; otherwise, start the non-blocking reception operation of the backpropagation calculation result of the M+1th micro-batch, and then proceed to the next step.

[0107] (4) Perform the backpropagation calculation of the Mth micro-batch based on the local model and the backpropagation calculation results of the target to obtain the corresponding backpropagation calculation results.

[0108] (5) Determine whether the current Mth micro-batch is the last micro-batch of the third stage.

[0109] (6) If so, the backpropagation calculation result of the Mth micro-batch is sent directly to the next higher-level computing node, and the pipeline operation is stopped.

[0110] (7) If not, start the non-blocking transmission operation of the backpropagation calculation result of the Mth micro-batch, then increment the back micro-batch number M, i.e., set M = M + 1, and return to execute the above step (2).

[0111] Furthermore, based on the above embodiments of the invention, the model training acceleration method provided in this embodiment further includes:

[0112] When the compute node is the first-level compute node in the pipeline, remove at least one of the following operations: receive forward propagation compute results, start a non-blocking receive operation for forward propagation compute results, wait for the non-blocking receive operation for forward propagation compute results to complete, send backward propagation compute results, start a non-blocking send operation for backward propagation compute results, and wait for the non-blocking send operation for backward propagation compute results to complete.

[0113] When the compute node is the last stage compute node in the pipeline, remove at least one of the following operations: send forward propagation compute results, start a non-blocking send operation of forward propagation compute results, wait for the non-blocking send operation of forward propagation compute results to complete, receive backward propagation compute results, start a non-blocking receive operation of backward propagation compute results, and wait for the non-blocking receive operation of backward propagation compute results to complete.

[0114] It should be understood that the receiving of forward propagation computation results mentioned in this embodiment refers to receiving the forward propagation computation results sent by the previous level computing node. If the current training process is the first level computing node, the following operations: receiving forward propagation computation results, starting the non-blocking reception operation of forward propagation computation results, and waiting for the non-blocking reception operation of forward propagation computation results to complete are no-operations, and related operations can be removed from the target training phase.

[0115] In this embodiment, sending the forward propagation calculation result refers to sending the forward propagation calculation result to the next level computing node. If the current training process is the last level computing node, the following operations: sending the forward propagation calculation result, starting the non-blocking sending operation of the forward propagation calculation result, and waiting for the non-blocking sending operation of the forward propagation calculation result to complete are no-operations, and the relevant operations can be removed from the target training stage.

[0116] The receiving of backpropagation computation results mentioned in this embodiment refers to receiving the backpropagation computation results sent by the next-level computing node. If the current training process is the last-level computing node, the following operations: receiving the backpropagation computation results, starting the non-blocking reception operation of the backpropagation computation results, and waiting for the non-blocking reception operation of the backpropagation computation results to complete are no-operations, and the relevant operations can be removed from the target training phase.

[0117] In this embodiment, sending the backpropagation calculation result refers to sending the backpropagation calculation result to the next higher-level computing node. If the current training process is the first-level computing node, the following operations: sending the backpropagation calculation result, starting the non-blocking sending operation of the backpropagation calculation result, and waiting for the non-blocking sending operation of the backpropagation calculation result to complete are no-operations, and the relevant operations can be removed from the target training phase.

[0118] Furthermore, based on the above embodiments of the invention, the non-blocking sending and non-blocking receiving involved in the embodiments of the present invention can be implemented by using a preset asynchronous communication interface, which includes at least the NVIDIA Collective Communication Library (NCCL) interface or the Message Passing (MPI) interface.

[0119] It is understood that the non-blocking mechanism in this embodiment of the invention uses the NCCL interface and MPI interface as asynchronous communication interfaces only as an example. In practical applications, other asynchronous non-blocking communication interfaces can also be used, as long as the same technical effect is achieved.

[0120] Example 2

[0121] Figure 7 This is a flowchart of a model training acceleration method provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment takes a GPU server as the computing node and provides an implementation method for accelerating model training. It enables overlapping execution of GPU communication and GPU computation, significantly reducing the requirements for network bandwidth and latency between GPU servers, reducing GPU computation waiting time, and thus improving model training efficiency and GPU computing power utilization. Figure 7 As shown in Embodiment 2 of the present invention, a model training acceleration method can be executed by a GPU server. The method specifically includes the following steps:

[0122] S510, Execute the initialization of the model training process, which includes pipelined parallelism.

[0123] S520. Determine the number of micro-batches corresponding to the first, second and third stages of the production line.

[0124] S530, Perform the first phase of computation and communication operations in a manner that overlaps computation and communication.

[0125] Specifically, such as Figure 8 As shown, S530 specifically includes the following steps:

[0126] S5301. Initialize the forward micro-batch number N and the backward micro-batch number M.

[0127] S5302, Receive the forward propagation calculation result of the Nth micro-batch.

[0128] S5303. Determine whether the current microbatch is the last microbatch in the current training iteration.

[0129] Specifically, if the current micro-batch is not the last micro-batch in the current training iteration, then execute S5304; otherwise, execute S5305.

[0130] S5304. Initiate the non-blocking reception operation of the forward propagation calculation results of the N+1th micro-batch.

[0131] S5305, Perform the forward propagation calculation for the Nth microbatch.

[0132] S5306. Determine whether the current micro-batch is the last micro-batch of the first stage.

[0133] Specifically, if the current micro-batch is the last micro-batch in the first stage, then execute S5307; otherwise, execute S5308.

[0134] S5307. Initiate the non-blocking transmission operation of the forward propagation calculation results of the Nth micro-batch and the non-blocking reception operation of the backward propagation calculation results of the Mth micro-batch. After the above non-blocking transmission and non-blocking reception operations are completed, the first stage ends.

[0135] S5308, Initiate the non-blocking transmission operation of the forward propagation calculation results of the Nth micro-batch.

[0136] S5309. Wait for the non-blocking receive operation of the forward propagation calculation result of the N+1th micro-batch to complete.

[0137] S5310, Let the forward micro-batch number N = N+1.

[0138] Specifically, after incrementing the forward micro-batch number N, the process returns to step S5303 above and repeats the loop until all micro-batches in the first stage have been executed.

[0139] It is worth noting that the number of times the loop is executed in the first phase is equal to the number of micro-batches in the first phase.

[0140] S540. If the number of micro-batches in the second stage is not 0, then the computation and communication operations of the second stage are performed in a manner that overlaps computation and communication.

[0141] Specifically, such as Figure 9 As shown, S540 specifically includes the following steps:

[0142] S5401, Waiting for the non-blocking receive operation of the forward propagation calculation result of the Nth micro-batch to complete.

[0143] S5402. Determine whether the current micro-batch is the first micro-batch of the second stage.

[0144] Specifically, if the current micro-batch is the first micro-batch of the second stage, then S5403 is executed; otherwise, S5404 is executed.

[0145] S5403: Initiate the non-blocking reception operation of the forward propagation calculation results of the N+1th micro-batch. After the above non-blocking reception operation is completed, proceed to S5407.

[0146] S5404. Determine whether the current micro-batch is the last micro-batch of the second stage.

[0147] Specifically, if the current micro-batch is not the last micro-batch in the stable phase, then execute S5405; otherwise, execute S5406.

[0148] S5405: Initiate the non-blocking transmission operation of the backpropagation calculation results of the (M-1)th micro-batch and the non-blocking reception operation of the forward propagation calculation results of the (N+1)th micro-batch. After the above non-blocking transmission and non-blocking reception operations are completed, proceed to S5407.

[0149] S5406. Initiate the non-blocking transmission operation of the backpropagation calculation results of the (M-1)th micro-batch. After the above non-blocking transmission operation is completed, proceed to S5407.

[0150] S5407, Perform the forward propagation calculation for the Nth microbatch.

[0151] S5408, Waiting for the non-blocking receive operation of the backpropagation calculation result of the Mth micro-batch to complete.

[0152] S5409, initiate the non-blocking transmission operation of the forward propagation calculation result of the Nth micro-batch, and the non-blocking reception operation of the backward propagation calculation result of the M+1th micro-batch.

[0153] S5410, Perform the backpropagation calculation for the Mth micro-batch.

[0154] S5411. Let the forward micro-batch number N = N+1 and the backward micro-batch number M = M+1.

[0155] S5412. Determine whether the current micro-batch is the last micro-batch of the second stage.

[0156] Specifically, if the current micro-batch is not the last micro-batch of the second stage, then return to execute S5401; otherwise, execute S5413.

[0157] S5413. Determine whether the number of micro-batches in the third stage is 0.

[0158] Specifically, if the number of micro-batches in the third stage is not 0, then S5414 is executed; otherwise, S5415 is executed.

[0159] S5414. Initiate the non-blocking transmission operation of the backpropagation calculation results of the (M-1)th micro-batch.

[0160] Specifically, if the number of micro-batches in the cooling phase is not zero, then the non-blocking transmission operation of the backpropagation calculation result of the M-1th micro-batch is started, the second phase ends, and then the third phase begins.

[0161] S5415, Send the backpropagation calculation results of the (M-1)th micro-batch.

[0162] Specifically, if the number of micro-batches in the third stage is 0, the backpropagation calculation result of the M-1th micro-batch is sent directly to the next higher-level computing node, and the pipeline operation is stopped.

[0163] It is worth noting that the number of times the second phase loop is executed is equal to the number of micro-batches in the second phase.

[0164] S550. If the number of micro-batches in the third stage is not 0, then the computation and communication operations of the third stage are performed in a manner that overlaps computation and communication.

[0165] Specifically, such as Figure 10 As shown, S550 specifically includes the following steps:

[0166] S5501, Waiting for the non-blocking receive operation of the backpropagation calculation result of the Mth micro-batch to complete.

[0167] S5502. Determine whether the current micro-batch is the last micro-batch of the third stage.

[0168] Specifically, if the current micro-batch is not the last micro-batch in the third stage, then S5503 is executed; otherwise, S5504 is executed.

[0169] S5503, Initiate the non-blocking reception operation of the backpropagation calculation results of the M+1th micro-batch.

[0170] S5504, Perform the backpropagation calculation for the Mth micro-batch.

[0171] S5505. Determine whether the current micro-batch is the last micro-batch of the third stage.

[0172] Specifically, if the current micro-batch is the last micro-batch in the third stage, then execute S5506; otherwise, execute S5507.

[0173] S5506. Send the backpropagation calculation result of the Mth micro-batch. After the above backpropagation calculation result is sent, the third stage ends.

[0174] S5507. Initiate the non-blocking transmission operation of the backpropagation calculation results of the Mth micro-batch.

[0175] S5508, set the subsequent micro-batch number M = M + 1.

[0176] Specifically, after incrementing the backward micro-batch number M, the process can return to step S5501 and execute it again.

[0177] It is worth noting that the number of times the third phase is executed is equal to the number of micro-batches in the third phase.

[0178] Furthermore, the execution processes of the first, second, and third stages described above are presented from the perspective of non-first and last nodes (i.e., not first-level or last-level computation nodes) participating in pipelined parallel training. For first-level computation nodes, the following operations need to be removed during the training process: receiving forward propagation computation results, initiating non-blocking forward propagation computation result reception, waiting for non-blocking forward propagation computation result reception to complete, sending backward propagation computation results, initiating non-blocking backward propagation computation result transmission, and waiting for non-blocking backward propagation computation result transmission to complete. For last-level computation nodes, the following operations need to be removed during the training process: sending forward propagation computation results, initiating non-blocking forward propagation computation result transmission, waiting for non-blocking forward propagation computation result transmission to complete, receiving backward propagation computation results, initiating non-blocking backward propagation computation result reception, and waiting for non-blocking backward propagation computation result reception to complete.

[0179] Figure 11 This is a timing diagram of a model training acceleration method provided in Embodiment 2 of the present invention. Figure 11 As shown, the pipeline has four stages, each requiring one GPU server; each model training iteration contains 16 micro-batches; blue squares represent forward propagation computation, and green squares represent backward propagation computation; the numbers in the blue / green squares represent the micro-batch numbers (from 1 to 16); the horizontal axis is the time axis. The pipeline execution process can be divided into three stages: the first stage, the second stage, and the third stage. Black arrows indicate the transmission of forward propagation computation results, and red arrows indicate the transmission of backward propagation computation results, with the numbers next to the black / red arrows representing the corresponding micro-batch numbers. Figure 11 It is easy to see that the model training acceleration method provided by the embodiments of the present invention can make most of the GPU communication and GPU computing in the pipeline overlap, which can greatly reduce the requirements for network bandwidth and latency between GPU servers, reduce the waiting time of GPU computing, and thus improve model training efficiency and GPU computing power utilization.

[0180] Example 3

[0181] Figure 12 This is a schematic diagram of a model training acceleration device provided in Embodiment 3 of the present invention. Figure 12 As shown, the device can be configured in a computing node, and specifically includes:

[0182] The model training initialization module 61 is used to perform the initialization of the model training process, which includes pipelined parallelism.

[0183] The computation and communication overlap execution module 62 is used to receive the forward propagation calculation result of the next micro-batch in non-blocking before executing the forward propagation calculation of the current micro-batch in the first stage of each pipeline, and to send the forward propagation calculation result of the current micro-batch in non-blocking after completing the forward propagation calculation of the current micro-batch, so as to realize the overlap execution of computation and communication.

[0184] Furthermore, based on the above embodiments of the invention, the execution process of the first stage includes:

[0185] Obtain the number of micro-batches corresponding to the first, second, and third stages of the pipeline, respectively;

[0186] Initialize the forward micro-batch number N and the backward micro-batch number M;

[0187] Receive the forward propagation calculation results of the target in the Nth micro-batch from the previous computing node;

[0188] If the current Nth microbatch is not the last microbatch in the current training iteration, then initiate the non-blocking reception operation of the forward propagation calculation result of the N+1th microbatch.

[0189] Execute the forward propagation calculation for the Nth microbatch based on the forward propagation calculation results of the target;

[0190] If the Nth micro-batch is not the last micro-batch in the first stage, then initiate the non-blocking transmission operation of the forward propagation calculation result of the Nth micro-batch, and after waiting for the non-blocking reception operation of the forward propagation calculation result of the N+1th micro-batch to be completed, increment the forward micro-batch number N, and return to the step of initiating the non-blocking reception operation of the forward propagation calculation result of the N+1th micro-batch if the current Nth micro-batch is not the last micro-batch in the current training iteration.

[0191] If the Nth micro-batch is the last micro-batch in the first stage, then the non-blocking transmission operation of the forward propagation calculation result of the Nth micro-batch is initiated, and the non-blocking reception operation of the backward propagation calculation result of the Mth micro-batch is initiated. The process will proceed to the second stage if the number of micro-batches in the second stage is not zero, or to the third stage if the number of micro-batches in the second stage is zero.

[0192] Furthermore, based on the above embodiments, the pipeline also includes a second stage, the execution flow of which includes:

[0193] Obtain the number of micro-batches corresponding to the second and third stages in the pipeline, and obtain the forward micro-batch number N and the backward micro-batch number M after the first stage;

[0194] Wait for the non-blocking reception operation of the target forward propagation calculation result of the Nth micro-batch to complete;

[0195] Based on the current position of the Nth micro-batch in the second stage, initiate the corresponding non-blocking receive operation or non-blocking send operation.

[0196] Execute the forward propagation calculation for the Nth microbatch based on the forward propagation calculation results of the target;

[0197] After the non-blocking reception operation of the backpropagation calculation result of the target of the Mth micro-batch is completed, the non-blocking transmission operation of the forward propagation calculation result of the Nth micro-batch is started, and the non-blocking reception operation of the backpropagation calculation result of the M+1th micro-batch is started.

[0198] Based on the backpropagation calculation results of the target, perform the backpropagation calculation of the Mth micro-batch, and increment the forward micro-batch number N and the backward micro-batch number M after the calculation is completed;

[0199] If the Nth micro-batch is not the last micro-batch of the second phase, return to the step of waiting for the non-blocking receive operation of the target forward propagation calculation result of the Nth micro-batch to complete.

[0200] If the Nth micro-batch is the last micro-batch of the second stage and the number of micro-batches in the third stage is not zero, then the non-blocking transmission operation of the backpropagation calculation result of the M-1th micro-batch is started, and the process proceeds to the third stage.

[0201] If the Nth micro-batch is the last micro-batch of the second stage and the number of micro-batches in the third stage is 0, then send the backpropagation calculation result of the M-1th micro-batch to the next higher-level computing node and stop the pipeline operation.

[0202] Furthermore, based on the above embodiments of the invention, according to the current position of the micro-batch in the stable phase, the corresponding non-blocking receive operation or non-blocking transmit operation is initiated, including:

[0203] If the Nth micro-batch is the first micro-batch of the second stage, then the non-blocking reception operation of the forward propagation calculation result of the N+1th micro-batch is initiated.

[0204] If the Nth micro-batch is not the first or last micro-batch of the second phase, then initiate the non-blocking transmission of the backpropagation calculation result of the M-1th micro-batch and initiate the non-blocking reception of the forward propagation calculation result of the N+1th micro-batch.

[0205] If the Nth micro-batch is the last micro-batch of the second phase, then the non-blocking transmission operation of the backpropagation calculation result of the (M-1)th micro-batch is initiated.

[0206] Furthermore, based on the above embodiments of the invention, the pipeline also includes a third stage, the execution flow of which includes:

[0207] Obtain the forward micro-batch number N and the backward micro-batch number M after the first stage, or the forward micro-batch number N and the backward micro-batch number M after the first and second stages.

[0208] Wait for the non-blocking reception operation of the backpropagation calculation results of the target in the Mth micro-batch to complete;

[0209] If the current Mth micro-batch is not the last micro-batch of the third stage, then initiate the non-blocking reception operation of the backpropagation calculation result of the M+1th micro-batch, perform the backpropagation calculation of the Mth micro-batch according to the target backpropagation calculation result, then initiate the non-blocking transmission operation of the backpropagation calculation result of the Mth micro-batch, increment the backpropagation micro-batch number M, and return to the step of waiting for the completion of the non-blocking reception operation of the target backpropagation calculation result of the Mth micro-batch.

[0210] If the Mth micro-batch is the last micro-batch in the third stage, then the backpropagation calculation of the Mth micro-batch is performed based on the target backpropagation calculation result, and the backpropagation calculation result of the Mth micro-batch is sent to the next higher-level computing node, and the pipeline operation is stopped.

[0211] Furthermore, based on the above embodiments of the invention, the model training acceleration device further includes:

[0212] The first operation removal module is used to remove at least one of the following operations when the computing node is the first-level computing node in the pipeline: receiving forward propagation computing results, starting a non-blocking receiving operation of forward propagation computing results, waiting for the non-blocking receiving operation of forward propagation computing results to complete, sending backward propagation computing results, starting a non-blocking sending operation of backward propagation computing results, and waiting for the non-blocking sending operation of backward propagation computing results to complete.

[0213] The second operation removal module is used to remove at least one of the following operations when the computing node is the last-stage computing node in the pipeline: sending forward propagation computing results, starting a non-blocking forward propagation computing result sending operation, waiting for the non-blocking forward propagation computing result sending operation to complete, receiving backward propagation computing results, starting a non-blocking backward propagation computing result receiving operation, and waiting for the non-blocking backward propagation computing result receiving operation to complete.

[0214] The model training acceleration device provided in the embodiments of the present invention can execute the model training acceleration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0215] Example 4

[0216] Figure 13 A schematic diagram of an electronic device 70 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0217] like Figure 13 As shown, the electronic device 70 includes at least one processor 71 and a memory, such as a read-only memory (ROM) 72 and a random access memory (RAM) 73, communicatively connected to the at least one processor 71. The memory stores computer programs executable by the at least one processor. The processor 71 can perform various appropriate actions and processes based on the computer program stored in the ROM 72 or loaded from storage unit 78 into the RAM 73. The RAM 73 can also store various programs and data required for the operation of the electronic device 70. The processor 71, ROM 72, and RAM 73 are interconnected via a bus 74. An input / output (I / O) interface 75 is also connected to the bus 74.

[0218] Multiple components in electronic device 70 are connected to I / O interface 75, including: input unit 76, such as keyboard, mouse, etc.; output unit 77, such as various types of monitors, speakers, etc.; storage unit 78, such as disk, optical disk, etc.; and communication unit 79, such as network card, modem, wireless transceiver, etc. Communication unit 79 allows electronic device 70 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0219] Processor 71 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 71 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 71 performs the various methods and processes described above, such as model training acceleration methods.

[0220] In some embodiments, the model training acceleration method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 78. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 70 via ROM 72 and / or communication unit 79. When the computer program is loaded into RAM 73 and executed by processor 71, one or more steps of the model training acceleration method described above may be performed. Alternatively, in other embodiments, processor 71 may be configured to execute the model training acceleration method by any other suitable means (e.g., by means of firmware).

[0221] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0222] In some embodiments, the model training acceleration method may be implemented as a computer program, which is implicitly included in a computer program product. When executed by a processor, the computer program implements the model training acceleration method of the present invention. The computer program product can be understood as a software product that primarily implements its solution through a computer program. The computer program used to implement the method of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer program causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer program may be executed entirely on a machine, partially on a machine, partially on a remote machine as a standalone software package, or entirely on a remote machine or server.

[0223] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0224] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0225] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0226] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0227] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0228] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for accelerating model training, characterized in that, The method, executed by a compute node, includes: Perform initialization of the model training process, which includes pipelined parallelism; In the first stage of each pipeline, before executing the forward propagation calculation of the current micro-batch, the forward propagation calculation result of the next micro-batch is received non-blockingly, and after completing the forward propagation calculation of the current micro-batch, the forward propagation calculation result of the current micro-batch is sent non-blockingly, so as to achieve overlapping execution of computation and communication.

2. The method according to claim 1, characterized in that, The execution process of the first stage includes: Obtain the number of micro-batches corresponding to the first stage, second stage and third stage in the pipeline, respectively; Initialize the forward micro-batch number N and the backward micro-batch number M; Receive the forward propagation calculation results of the target in the Nth micro-batch from the previous computing node; If the current Nth microbatch is not the last microbatch in the current training iteration, then initiate the non-blocking reception operation of the forward propagation calculation result of the N+1th microbatch. The forward propagation calculation of the Nth micro-batch is performed based on the forward propagation calculation result of the target; If the Nth micro-batch is not the last micro-batch in the first stage, then initiate the non-blocking transmission operation of the forward propagation calculation result of the Nth micro-batch, and after waiting for the non-blocking reception operation of the forward propagation calculation result of the (N+1)th micro-batch to be completed, increment the forward micro-batch number N, and return to execute the step of initiating the non-blocking reception operation of the forward propagation calculation result of the (N+1)th micro-batch if the current Nth micro-batch is not the last micro-batch in the current training iteration; If the Nth micro-batch is the last micro-batch in the first stage, then a non-blocking transmission operation of the forward propagation calculation result of the Nth micro-batch is initiated, and a non-blocking reception operation of the backward propagation calculation result of the Mth micro-batch is initiated. The process then proceeds to the second stage when the number of micro-batches in the second stage is not zero, or proceeds to the third stage when the number of micro-batches in the second stage is zero.

3. The method according to claim 1, characterized in that, The pipeline also includes a second stage, the execution flow of which includes: Obtain the number of micro-batches corresponding to the second stage and the third stage in the pipeline, respectively, and obtain the forward micro-batch number N and the backward micro-batch number M after the first stage; Wait for the non-blocking reception operation of the target forward propagation calculation result of the Nth micro-batch to complete; Based on the current position of the Nth micro-batch in the second stage, initiate the corresponding non-blocking receive operation or non-blocking send operation. The forward propagation calculation of the Nth micro-batch is performed based on the forward propagation calculation result of the target; After the non-blocking reception operation of the backpropagation calculation result of the target of the Mth micro-batch is completed, the non-blocking transmission operation of the forward propagation calculation result of the Nth micro-batch is started, and the non-blocking reception operation of the backpropagation calculation result of the M+1th micro-batch is started. The backward propagation calculation of the Mth micro-batch is performed based on the backpropagation calculation result of the target, and the forward micro-batch number N and the backward micro-batch number M are incremented after the calculation is completed. If the Nth micro-batch is not the last micro-batch of the second stage, then return to the step of waiting for the non-blocking receive operation of the target forward propagation calculation result of the Nth micro-batch to complete; If the Nth micro-batch is the last micro-batch of the second stage and the number of micro-batches in the third stage is not 0, then the non-blocking transmission operation of the backpropagation calculation result of the M-1th micro-batch is started, and the process proceeds to the third stage. If the Nth micro-batch is the last micro-batch of the second stage and the number of micro-batches in the third stage is 0, then the backpropagation calculation result of the (M-1)th micro-batch is sent to the next higher-level computing node, and the pipeline operation is stopped.

4. The method according to claim 3, characterized in that, The step of initiating the corresponding non-blocking receive operation or non-blocking transmit operation according to the current position of the Nth micro-batch in the second stage includes: If the Nth micro-batch is the first micro-batch of the second stage, then the non-blocking reception operation of the forward propagation calculation result of the (N+1)th micro-batch is initiated. If the Nth micro-batch is not the first micro-batch or the last micro-batch of the second stage, then the non-blocking transmission operation of the backpropagation calculation result of the M-1th micro-batch and the non-blocking reception operation of the forward propagation calculation result of the N+1th micro-batch are initiated. If the Nth micro-batch is the last micro-batch of the second stage, then the non-blocking transmission operation of the backpropagation calculation result of the (M-1)th micro-batch is initiated.

5. The method according to claim 1, characterized in that, The pipeline also includes a third stage, the execution flow of which includes: Obtain the forward micro-batch number N and the backward micro-batch number M after the first stage, or the forward micro-batch number N and the backward micro-batch number M after the first stage and the second stage. Wait for the non-blocking reception operation of the backpropagation calculation results of the target in the Mth micro-batch to complete; If the current Mth micro-batch is not the last micro-batch of the third stage, then the non-blocking reception operation of the backpropagation calculation result of the (M+1)th micro-batch is initiated, and the backpropagation calculation of the Mth micro-batch is performed according to the target backpropagation calculation result. Then, the non-blocking transmission operation of the backpropagation calculation result of the Mth micro-batch is initiated, the backpropagation micro-batch number M is incremented, and the process returns to the step of waiting for the completion of the non-blocking reception operation of the target backpropagation calculation result of the Mth micro-batch. If the Mth micro-batch is the last micro-batch of the third stage, then the backpropagation calculation of the Mth micro-batch is performed according to the target backpropagation calculation result, and the backpropagation calculation result of the Mth micro-batch is sent to the next higher-level computing node, and the pipeline operation is stopped.

6. The method according to any one of claims 2-5, characterized in that, Also includes: When the computing node is the first-level computing node in the pipeline, remove at least one of the following operations: receiving forward propagation computing results, initiating a non-blocking reception operation for forward propagation computing results, waiting for the non-blocking reception operation for forward propagation computing results to complete, sending backward propagation computing results, initiating a non-blocking transmission operation for backward propagation computing results, and waiting for the non-blocking transmission operation for backward propagation computing results to complete. When the computing node is the last-stage computing node in the pipeline, remove at least one of the following operations: sending forward propagation computing results, initiating a non-blocking forward propagation computing result sending operation, waiting for the non-blocking forward propagation computing result sending operation to complete, receiving backward propagation computing results, initiating a non-blocking backward propagation computing result receiving operation, and waiting for the non-blocking backward propagation computing result receiving operation to complete.

7. A model training acceleration device, characterized in that, Configured in a compute node, the device includes: The model training initialization module is used to perform the initialization of the model training process, which includes pipelined parallelism. The computation and communication overlapped execution module is used to receive the forward propagation calculation result of the next micro-batch in non-blocking before executing the forward propagation calculation of the current micro-batch in the first stage of each pipeline, and to send the forward propagation calculation result of the current micro-batch in non-blocking after completing the forward propagation calculation of the current micro-batch, so as to realize the overlapped execution of computation and communication.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the model training acceleration method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the model training acceleration method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the model training acceleration method according to any one of claims 1-6.

Images