Load regulation and control method and device based on staggered assembly line parallelism

Abstract

The invention relates to a load regulation and control method and device based on staggered assembly line parallelism. The method comprises the following steps: determining allocation parameters required for resource allocation in a target model training process, wherein the allocation parameters comprise the number of computational nodes participating in target model training, a pipeline parallel scale and the number of virtual stages required to be executed in each computational node of a target model; determining a calculation strategy of each calculation node for each micro-batch according to the distribution parameters and the node number of each calculation node, wherein the calculation strategy comprises a calculation type needing to execute calculation and a preheating calculation quantity; and controlling each calculation node to perform calculation of each micro-batch according to the corresponding calculation strategy so as to perform training aiming at the target model until the training is completed. Through the fine load balancing design, the efficiency of distributed training is improved, sustainable benefits in economy and environment are brought, and the method has wide application prospects in commercial and academic research.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a load control method and device based on staggered pipeline parallelism. Background Art

[0002] In distributed training systems, the pipeline parallelism mode is one of the common methods to improve model training efficiency. This mode is achieved by deploying different parts of the model on different computing nodes. Although this can accelerate model training to a certain extent, its core problem lies in the uneven distribution of computing load, especially in the first and last stages of the pipeline. This imbalance leads to a waste of computing resources, because some processors may be idle while waiting for other stages to complete tasks, making resource allocation inefficient, increasing training time and cost, and making it a technical problem that needs to be solved urgently. Summary of the invention

[0003] In view of this, the present disclosure proposes a load control method and device based on staggered pipeline parallelism.

[0004] According to one aspect of the present disclosure, a load control method based on staggered pipeline parallelism is provided, which is applied to distributed training of a target model, and the method includes:

[0005] Determine allocation parameters required for resource allocation during training of the target model, wherein the allocation parameters include the number of computing nodes participating in the training of the target model, the pipeline parallel scale, and the number of virtual stages to be executed in each computing node of the target model;

[0006] Determining a computing strategy of each computing node for each micro-batch according to the allocation parameters and the node number of each computing node, wherein the computing strategy includes a computing type to be performed and a number of preheating calculations;

[0007] Control each of the computing nodes to perform calculations on each of the micro-batches according to a corresponding computing strategy to perform training for the target model until the training is completed.

[0008] In a possible implementation, the calculation type corresponding to each of the computing nodes includes forward calculation and reverse propagation calculation, and the calculation type corresponding to the 1 / 2 computing nodes with the node numbers ranked in front among the computing nodes also includes word embedding calculation and word embedding reverse calculation; the calculation type corresponding to the 1 / 2 computing nodes with the node numbers ranked in the back among the computing nodes also includes output calculation and output reverse calculation.

[0009] In a possible implementation, determining a computing strategy of each computing node for each micro-batch according to the allocation parameter and the node number of each computing node includes at least one of the following operations:

[0010] Determine the number of preheating calculations corresponding to each computing node according to the parallel scale of the pipeline, the node number of each computing node, and the number of virtual stages;

[0011] Determining, according to the batch number of each micro-batch and the parallel scale of the pipeline, a computing node for executing word embedding calculations of each micro-batch;

[0012] Determining, according to the batch number of each micro-batch and the parallel scale of the pipeline, a computing node for performing word embedding reverse calculation of each micro-batch;

[0013] According to the batch number of each micro-batch and the parallel scale of the pipeline, a computing node for executing output calculation of each micro-batch and a computing node for executing output reverse calculation of each micro-batch are determined.

[0014] In a possible implementation, determining the number of preheating calculations corresponding to each computing node according to the parallel scale of the pipeline, the node number of each computing node, and the number of the virtual stages includes:

[0015] Subtract the node number of the computing node and 1 from the pipeline parallel scale in sequence, and then multiply the result by 2 to obtain a first result; subtract 1 from the number of virtual stages required to be executed by the computing node, and then multiply the result by the pipeline parallel scale to obtain a second result; and use the sum of the first result and the second result as the number of preheating calculations for the corresponding computing node;

[0016] The node sequence number is the value of the node number minus 1.

[0017] In a possible implementation, controlling each computing node to perform the calculation of each micro-batch according to a corresponding computing strategy includes:

[0018] When it is determined that the number of forward calculations performed by the computing node reaches the number of preheating calculations corresponding to the computing node, it is determined that the computing node completes the preheating stage calculation, and the computing node is controlled to start one forward and one reverse 1F1B stage calculations.

[0019] In a possible implementation, determining a computing node for executing word embedding calculations for each micro-batch according to the batch number of each micro-batch and the parallel scale of the pipeline includes:

[0020] For a micro-batch whose batch number is greater than or equal to 2 times the parallel scale of the pipeline, a first remainder is obtained by taking the difference of the batch number of the micro-batch minus 1 modulo the parallel scale of the pipeline, and a second remainder obtained by taking the first remainder modulo half of the parallel scale of the pipeline plus 1 as the word embedding node number corresponding to the batch number;

[0021] The computing node corresponding to the node number and the word embedding node number is determined as the computing node for performing the word embedding calculation of the corresponding micro-batch.

[0022] In a possible implementation, controlling each computing node to perform the calculation of each micro-batch according to a corresponding computing strategy includes:

[0023] In a case where it is determined that the computing node that performs the word embedding calculation of the micro-batch is not the first computing node, the micro-batch that completes the word embedding calculation is transferred to the first computing node.

[0024] In a possible implementation, determining a computing node for performing word embedding reverse calculation of each micro-batch according to the batch number of each micro-batch and the parallel scale of the pipeline includes:

[0025] When the remainder obtained by taking the batch number of the micro-batch modulo half of the pipeline parallel scale is 0, it is determined that word embedding reverse calculation of the micro-batch needs to be performed; and the difference after the batch number of the micro-batch is subtracted by 1 is modulo half of the pipeline parallel scale to obtain a third remainder, and the difference obtained by subtracting the third remainder from half of the pipeline parallel scale is used as the word embedding reverse node number corresponding to the batch number; the computing node corresponding to the node number and the word embedding reverse node number is determined as the computing node for performing word embedding reverse calculation of the micro-batch, so that the word embedding reverse calculation of the micro-batch is performed after the corresponding word embedding calculation.

[0026] In a possible implementation, determining a computing node for executing output calculation of each micro-batch and a computing node for executing output reverse calculation of each micro-batch according to the batch number of each micro-batch and the parallel scale of the pipeline includes:

[0027] When the remainder obtained by taking the batch number of the micro-batch modulo half of the pipeline parallel scale is not 0, the sum of the quotient obtained by dividing the pipeline parallel scale by 2 and the remainder obtained by taking the batch number modulo half of the pipeline parallel scale is determined as the output node number of the batch number;

[0028] Determine a computing node corresponding to a node number and the output node number as a computing node that performs output calculation of the micro-batch and subsequent output reverse calculation;

[0029] The starting condition for the computing node corresponding to each output node number to execute the output calculation of the micro-batch and the subsequent output reverse calculation is that the remainder obtained by taking the batch number modulo half of the parallel scale of the pipeline is 0.

[0030] According to another aspect of the present disclosure, a load control device based on staggered pipeline parallelism is provided, which is applied to distributed training of a target model, and the device includes:

[0031] A parameter determination module, used to determine the allocation parameters required for resource allocation during the training of the target model, wherein the allocation parameters include the number of computing nodes participating in the training of the target model, the pipeline parallel scale, and the number of virtual stages to be executed in each computing node of the target model;

[0032] A computing strategy determination module, used to determine the computing strategy of each computing node for each micro-batch according to the allocation parameters and the node number of each computing node, wherein the computing strategy includes the type of computing to be performed and the number of preheating computing;

[0033] The computing control module is used to control each computing node to perform the calculation of each micro-batch according to the corresponding computing strategy to perform training for the target model until the training is completed.

[0034] In a possible implementation, the calculation type corresponding to each of the computing nodes includes forward calculation and reverse propagation calculation, and the calculation type corresponding to the 1 / 2 computing nodes with the node numbers ranked in front among the computing nodes also includes word embedding calculation and word embedding reverse calculation; the calculation type corresponding to the 1 / 2 computing nodes with the node numbers ranked in the back among the computing nodes also includes output calculation and output reverse calculation.

[0035] In a possible implementation, the computing strategy determination module includes at least one of the following submodules:

[0036] A preheating calculation submodule, used to determine the number of preheating calculations corresponding to each computing node according to the parallel scale of the pipeline, the node number of each computing node and the number of virtual stages;

[0037] A word embedding calculation submodule, used to determine the computing node for executing the word embedding calculation of each micro-batch according to the batch number of each micro-batch and the parallel scale of the pipeline;

[0038] A word embedding reverse calculation submodule, used to determine the computing node for performing the word embedding reverse calculation of each micro-batch according to the batch number of each micro-batch and the parallel scale of the pipeline;

[0039] The output calculation submodule is used to determine the computing nodes that perform the output calculation of each micro-batch and the computing nodes that perform the output reverse calculation of each micro-batch according to the batch number of each micro-batch and the parallel scale of the pipeline.

[0040] In a possible implementation, determining the number of preheating calculations corresponding to each computing node according to the parallel scale of the pipeline, the node number of each computing node, and the number of the virtual stages includes:

[0041] Subtract the node number of the computing node and 1 from the pipeline parallel scale in sequence, and then multiply the result by 2 to obtain a first result; subtract 1 from the number of virtual stages required to be executed by the computing node, and then multiply the result by the pipeline parallel scale to obtain a second result; and use the sum of the first result and the second result as the number of preheating calculations for the corresponding computing node;

[0042] The node sequence number is the value of the node number minus 1.

[0043] In a possible implementation, the computing control module includes:

[0044] The first stage control submodule is used to determine that the computing node has completed the preheating stage calculation and control the computing node to start one forward and one reverse 1F1B stage calculation when it is determined that the number of forward calculations performed by the computing node reaches the preheating calculation number corresponding to the computing node.

[0045] In a possible implementation, determining a computing node for executing word embedding calculations for each micro-batch according to the batch number of each micro-batch and the parallel scale of the pipeline includes:

[0046] For a micro-batch whose batch number is greater than or equal to 2 times the parallel scale of the pipeline, a first remainder is obtained by taking the difference of the batch number of the micro-batch minus 1 modulo the parallel scale of the pipeline, and a second remainder obtained by taking the first remainder modulo half of the parallel scale of the pipeline plus 1 as the word embedding node number corresponding to the batch number;

[0047] The computing node corresponding to the node number and the word embedding node number is determined as the computing node for performing the word embedding calculation of the corresponding micro-batch.

[0048] In a possible implementation, the computing control module includes:

[0049] A transfer submodule is used to transfer the micro-batch that completes the word embedding calculation to the first computing node when it is determined that the computing node that performs the word embedding calculation of the micro-batch is not the first computing node.

[0050] In a possible implementation, determining a computing node for performing word embedding reverse calculation of each micro-batch according to the batch number of each micro-batch and the parallel scale of the pipeline includes:

[0051] When the remainder obtained by taking the batch number of the micro-batch modulo half of the pipeline parallel scale is 0, it is determined that word embedding reverse calculation of the micro-batch needs to be performed; and the difference after the batch number of the micro-batch is subtracted by 1 is modulo half of the pipeline parallel scale to obtain a third remainder, and the difference obtained by subtracting the third remainder from half of the pipeline parallel scale is used as the word embedding reverse node number corresponding to the batch number; the computing node corresponding to the node number and the word embedding reverse node number is determined as the computing node for performing word embedding reverse calculation of the micro-batch, so that the word embedding reverse calculation of the micro-batch is performed after the corresponding word embedding calculation.

[0052] In a possible implementation, determining a computing node for executing output calculation of each micro-batch and a computing node for executing output reverse calculation of each micro-batch according to the batch number of each micro-batch and the parallel scale of the pipeline includes:

[0053] When the remainder obtained by taking the batch number of the micro-batch modulo half of the pipeline parallel scale is not 0, the sum of the quotient obtained by dividing the pipeline parallel scale by 2 and the remainder obtained by taking the batch number modulo half of the pipeline parallel scale is determined as the output node number of the batch number;

[0054] Determine a computing node corresponding to a node number and the output node number as a computing node that performs output calculation of the micro-batch and subsequent output reverse calculation;

[0055] The starting condition for the computing node corresponding to each output node number to execute the output calculation of the micro-batch and the subsequent output reverse calculation is that the remainder obtained by taking the batch number modulo half of the parallel scale of the pipeline is 0.

[0056] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0057] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0058] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0059] The load control method and device based on staggered pipeline parallelism provided by the embodiments of the present disclosure not only improve the efficiency of distributed training through sophisticated load balancing design, but also bring economic and environmental sustainability benefits, and have broad application prospects in both business and academic research.

[0060] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0062] Figure 1 A flow chart of a load control method based on staggered pipeline parallelism according to an embodiment of the present disclosure is shown.

[0063] Figure 2 A schematic diagram showing Example 1 of distributed training according to an embodiment of the present disclosure is shown.

[0064] Figure 3 It is a block diagram of a device 1900 for load regulation based on staggered pipeline parallelism according to an exemplary embodiment. DETAILED DESCRIPTION

[0065] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0066] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0067] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0068] In the related art, the pipeline parallel mode in the distributed training system has the problem of uneven distribution of computing load, especially in the first and last stages of the pipeline. Specifically, the processors in the first stage (i.e. the head stage) usually bear the additional load of data preprocessing and word embedding calculation, while the processors in the last stage (i.e. the tail stage) need to process the output layer (lm_head layer or classifier layer) and the additional load of loss function calculation. This imbalance leads to a waste of computing resources, inefficient resource allocation, and increased training time and cost.

[0069] In the related art, although the pipeline parallel mode can theoretically improve the efficiency of distributed training, there are some significant technical disadvantages in practice. First, the computing nodes face the problem of load imbalance when processing tasks in the first and last stages. Specifically, the first stage of the pipeline is usually overloaded because it needs to process data acquisition (including data preprocessing) and word embedding calculations at the same time, while the last stage is also overloaded because it needs to process a large number of output layer calculations and loss function calculations. This load imbalance not only reduces computing efficiency, but also leads to a waste of resources, because processors in other stages may be idle while waiting for the completion of the first and last stages.

[0070] In addition, the related technologies lack an effective load regulation mechanism to dynamically balance the workload at each stage, which limits the scalability and flexibility of the entire system. This problem is particularly evident when processing large-scale data, because it is impossible to adjust resource allocation according to actual computing needs, thus affecting the overall training efficiency and cost-effectiveness.

[0071] To solve the above technical problems, the embodiments of the present disclosure provide a load control method and device based on interleaving pipeline parallelism. Through sophisticated load balancing design, not only the efficiency of distributed training is improved, but also economic and environmental sustainability benefits are brought, which has broad application prospects in both business and academic research.

[0072] like Figure 1 As shown, the method includes steps S101 to S103. The method is applied to distributed training for a target model. The distributed training for the target model is performed in a computing device having multiple computing nodes. The main computing node among the multiple computing nodes, the controller in the computing device, or the processor in the computing device is used to execute the method provided by the embodiment of the present disclosure, which can improve the efficiency of distributed training of the computing device.

[0073] In step S101, allocation parameters required for resource allocation in the process of training the target model are determined, and the allocation parameters may include various parameter information related to computing nodes and target models used as a basis for load regulation. In some embodiments, the allocation parameters may include the number of computing nodes (devices) participating in the training of the target model, the pipeline parallel size, and the number of virtual stages that need to be executed in each computing node of the target model.

[0074] In this embodiment, the difference in model size such as the number of layers of the target model, the difference in computing node related information such as the number of computing nodes and the performance of each computing node, and the pipeline parallel scale in distributed training affect the number of virtual stages that need to be executed in each computing node. The pipeline parallel scale can be the number of micro-batches that each computing node can process simultaneously.

[0075] For example, assuming that the target model A can have 16 transformer layers, if the number of virtual stages to be executed in each computing node is 2 and the number of nodes is 4, then the 16 layers 1, 2, 9, and 10 will be assigned to the first computing node, the 3, 4, 11, and 12 layers will be assigned to the second computing node, the 5, 6, 13, and 14 layers will be assigned to the third computing node, and the 7, 8, 15, and 16 layers will be assigned to the fourth computing node. Each computing node includes two virtual stages, virtual stage 0 and virtual stage 1, which calculate two layers respectively. Layers 1, 2, 9, and 10 are assigned to the first computing node. Virtual stage 0 is used to calculate the 1st and 2nd layers, virtual stage 1 is used to calculate the 9th and 10th layers, and so on. During the forward calculation (forward stage), first calculate the 1st and 2nd layers (virtual stage 0) on the first computing node, and then transfer them to the second computing node to calculate the 3rd and 4th layers (virtual stage 0). After completion, transfer them to the third computing node to calculate the 5th and 6th layers (virtual stage 0), and so on until the 7th and 8th layers (virtual stage 0) on the fourth computing node are calculated. After completion, they are transferred back to the first computing node to calculate the 9th and 10th layers (virtual stage 0), and the 16-layer forward calculation is completed in sequence. The backward stage is the opposite process. Assume that the target model A can have 16 transformer layers. If the number of virtual stages to be executed in each computing node is 4 and the number of nodes is 4, then the 1st, 5th, 9th, and 13th layers of the 16 layers will be allocated to the first computing node, the 2nd, 6th, 10th, and 14th layers will be allocated to the second computing node, the 3rd, 7th, 11th, and 15th layers will be allocated to the third computing node, and the 4th, 8th, 12th, and 16th layers will be allocated to the fourth computing node. Each computing node includes 4 virtual stages to perform calculations for 1 layer each.

[0076] In step S102, a computing strategy of each computing node for each micro-batch is determined according to the allocation parameters and the node number of each computing node, wherein the computing strategy includes a computing type to be performed and a preheating computing quantity.

[0077] In step S103, each of the computing nodes is controlled to perform calculations on each of the micro-batches according to a corresponding computing strategy to perform training for the target model until the training is completed.

[0078] In this embodiment, the calculation types corresponding to each of the computing nodes include forward calculation (also known as forward stage) and backward propagation calculation (also known as backward stage). The calculation types corresponding to the first 1 / 2 of the computing nodes in the computing nodes in the front node numbering also include word embedding calculation and word embedding reverse calculation; the calculation types corresponding to the last 1 / 2 of the computing nodes in the computing nodes in the back node numbering also include output calculation and output reverse calculation. Among them, word embedding calculation may include data preprocessing and word embedding calculation, and data preprocessing may include data cleaning, normalization, encoding, segmentation and other processing; word embedding reverse calculation is the reverse calculation of word embedding calculation. Output calculation may include output layer calculation and loss function calculation, and output reverse calculation may be the reverse calculation of output calculation. In this way, each computing node can perform forward and backward propagation calculations normally. At the same time, the word embedding calculations and word embedding reverse calculations originally performed by the first computing node are evenly divided into the first half of the computing nodes, and the output calculations and output reverse calculations originally performed by the last computing node are evenly divided into the last half of the computing nodes. This achieves balanced regulation of the computing node workload, maximizes the utilization of the computing resources of all computing nodes, and achieves resource optimization.

[0079] In the present embodiment, a batch is a set of training samples for a round of training during model training, which can be obtained by splitting the entire set of training samples. Each batch includes multiple training samples, each of which is independent of each other and participates in the same round of training process. When all the training samples in a batch are calculated, it is said that one round of training for the model is completed. In order to speed up the speed of model training, each batch is split into multiple microbatches in the present disclosure. Among them, a microbatch is a smaller batch obtained by further splitting the training samples in the batch. In short, a microbatch is a subset of a batch, and a batch includes multiple microbatches, and different microbatches belonging to the same batch participate in the same round of training process. A microbatch is the smallest unit of calculation for a computing node to perform a forward calculation, a backward calculation, and other different types of calculations.

[0080] For example, if Figure 2Example 1 shown is a schematic diagram of distributed training for a target model according to a load control method based on staggered pipeline parallelism provided in an embodiment of the present disclosure. In this example 1, the computing nodes include 4, namely Device 1, Device 2, Device 3 and Device 4. The number of virtual stages executed by each computing node is 2, namely virtual stage 0 and virtual stage 1. Each number represents the calculation of different types of calculations performed by a micro-batch with a batch number of that number. Then, Device 1 and Device 2 can be controlled to perform word embedding calculations and word embedding reverse calculations, and Device 3 and Device 4 can be controlled to perform output calculations and output reverse calculations, thereby achieving a balanced distribution of loads in the first and last stages, and significantly reducing the bubbles of each computing node.

[0081] In a possible implementation, step S102 may include at least one of the following operations.

[0082] Operation 1: determining the number of preheating calculations corresponding to each computing node according to the parallel scale of the pipeline, the node number of each computing node, and the number of the virtual stages.

[0083] In this embodiment, the number of warm-up calculations may refer to the number of forward calculations that each computing node needs to perform in the warm-up phase (Warmup). Among them, the warm-up phase of each computing node may refer to the phase in which the computing node only performs forward calculations in the initial stage of target model training, and the computing node does not enter the forward and reverse 1F1B phase at this stage. The node number of the computing node is used to indicate the ranking of the corresponding computing node among all computing nodes, and the number may be a positive integer such as 1 or 2. The smaller the number in the front ranking, the larger the number in the back ranking. For example, if there are 4 computing nodes, the node numbers from the first computing node to the fourth computing node may be 1, 2, 3, and 4, respectively.

[0084] In some embodiments, the first result can be obtained by subtracting the node number of the computing node and 1 from the pipeline parallel scale and then multiplying by 2, the second result can be obtained by subtracting 1 from the number of virtual stages required to be executed by the computing node and then multiplying by the pipeline parallel scale, and the sum of the first result and the second result can be used as the number of preheating calculations for the corresponding computing node. The node number (pipeline parallel rank) of the computing node can be the value of the node number minus 1. For example, if there are 4 computing nodes, the node numbers from the first computing node to the fourth computing node can be 1, 2, 3, and 4, respectively, and the node numbers can be 0, 1, 2, and 3, respectively.

[0085] Among them, the preheating calculation quantity can be calculated by the following formula:

[0086] num_warmup_microbatches=(pipeline_parallel_size-pipeline_parallel_rank-1)*2+(num_model_chunks-1)*pipeline_parallel_size

[0087] Among them, num_warmup_microbatches represents the number of warmup calculations for the computing node with the node number pipeline_parallel_rank. pipeline_parallel_size is the pipeline parallel scale, and num_model_chunks is the number of virtual stages that the computing node needs to execute. Among them, the node number is a non-negative integer.

[0088] Based on the method for determining the number of preheating calculations provided by operation 1, step S103 may include: when it is determined that the number of forward calculations performed by the computing node reaches the number of preheating calculations corresponding to the computing node, determining that the computing node completes the preheating stage calculation, and controlling the computing node to start one forward and one reverse 1F1B stage calculations.

[0089] For example, if Figure 2 In Example 1 shown in the figure, there are 4 computing nodes participating in the target model training. These 4 computing nodes are Device1, Device2, Device3 and Device4. The corresponding node numbers (pipeline_parallel_rank) can be 0, 1, 2, and 3 respectively. According to the above formula, the number of warm-up calculations corresponding to the 4 computing nodes is as follows: Figure 2 The numbers shown are 10, 8, 6, and 4 respectively.

[0090] Operation 2: Determine a computing node for executing word embedding calculations for each of the micro-batches according to the batch number of each of the micro-batches and the parallel scale of the pipeline.

[0091] In this embodiment, the batch number of micro-batches can be set according to the order in which each micro-batch is processed. For example, the batch number of the first processed micro-batch is 1, the batch number of the second processed micro-batch is 2, and so on.

[0092] In some embodiments, the implementation of operation 2 may include: taking the difference of the batch number of the micro-batch minus 1 modulo the pipeline parallel scale to obtain a first remainder, taking the first remainder modulo half of the pipeline parallel scale to obtain a second remainder plus 1 as the word embedding node number corresponding to the batch number; and determining the computing node corresponding to the node number and the word embedding node number as the computing node that performs the word embedding calculation of the corresponding micro-batch.

[0093] Among them, for micro-batches with batch number num_of_microbatch ≥ 2*pipline_parallel_size, the computing node that performs word embedding calculations for each micro-batch can be calculated by the following formula:

[0094] The word embedding node number corresponding to the number of batches (Num_of_microbatch) = ((num_of_microbatch-1)% pipeline_parallel_size)% (pipeline_parallel_size / 2)+1

[0095] For example, for Figure 2 In the example 1 shown, when Num_of_microbatch is 9, ((9-1)%2)%2+1=1, and the word embedding calculation of the 9th micro-batch is performed by the computing node with node number 1; when Num_of_microbatch is 10, ((10-1)%2)%2+1=2, and the word embedding calculation of the 10th micro-batch is performed by the computing node with node number 2; when Num_of_microbatch is 11, ((11-1)%2)%2+1=1, and the word embedding calculation of the 11th micro-batch is performed by the computing node with node number 1; when Num_of_microbatch is 12, ((12-1)%2)%2+1=2, and the word embedding calculation of the 12th micro-batch is performed by the computing node with node number 2, and so on.

[0096] In some embodiments, based on the setting of operation 2, step S103 may include: when it is determined that the computing node that performs the word embedding calculation of the micro-batch is not the first computing node, the micro-batch that completes the word embedding calculation is transferred to the first computing node. In this way, it can be ensured that the word embedding calculation is not all performed in the first computing node, and the result can also be transferred back to the first computing node, so as to ensure the normal and orderly progress of the entire model training.

[0097] Operation 3: Determine a computing node for performing word embedding reverse calculation of each micro-batch according to the batch number of each micro-batch and the parallel scale of the pipeline.

[0098] In some embodiments, the implementation of operation 3 may include: when the remainder obtained by taking the batch number of the micro-batch modulo half of the pipeline parallel scale is 0, determining that the word embedding reverse calculation of the micro-batch needs to be performed; and taking the difference after the batch number of the micro-batch is subtracted by 1 modulo half of the pipeline parallel scale to obtain a third remainder, and subtracting the third remainder from half of the pipeline parallel scale to obtain the difference as the word embedding reverse node number corresponding to the batch number; determining the computing node corresponding to the node number and the word embedding reverse node number as the computing node for performing the word embedding reverse calculation of the micro-batch, so that the word embedding reverse calculation of the micro-batch is performed after the corresponding word embedding calculation. And the word embedding reverse calculation can also be assigned to the first half of the computing nodes for execution.

[0099] Among them, the following formula can be used to determine whether to start the word embedding reverse calculation:

[0100] K1=(num_of_microbatch)%(pipeline_parallel_size / 2)

[0101] If K1==0, it is determined that the word embedding reverse calculation can be started, otherwise, it is determined that the word embedding reverse calculation will not be performed. For example, if 1%(4 / 2)=1, the word embedding reverse calculation will not be performed, and if 2%(4 / 2)==0, the word embedding reverse calculation will be started.

[0102] Wherein, when it is determined to start the word embedding reverse calculation, the word embedding reverse node number of the computing node that performs the word embedding reverse calculation of the micro-batch can be calculated by the following formula:

[0103] (pipeline_parallel_size / 2)-(num_of_microbatch-1)%(pipeline_parallel_size / 2).

[0104] For example, if Figure 2 In Example 1 shown, the word embedding calculations of the red boxes performed by Device1 are the word embedding calculations corresponding to micro-batches with batch numbers of 7, 8, 9, and 11, and the word embedding calculations of the red boxes performed by Device2 are the word embedding calculations corresponding to micro-batches with batch numbers of 10 and 12. The word embedding reverse calculations of the orange boxes performed by Device1 are the word embedding reverse calculations corresponding to micro-batches with batch numbers of 2 and 4, and the word embedding reverse calculations of the orange boxes performed by Device2 are the word embedding reverse calculations corresponding to micro-batches with batch numbers of 1 and 3.

[0105] Operation 4: determining, according to the batch number of each micro-batch and the parallel scale of the pipeline, a computing node for executing output calculation of each micro-batch and a computing node for executing output reverse calculation of each micro-batch.

[0106] In some embodiments, the implementation of operation 4 may include: when the remainder obtained by taking the batch number of the micro-batch modulo half of the pipeline parallel scale is not 0, the sum of the quotient obtained by dividing the pipeline parallel scale by 2 and the remainder obtained by taking the batch number modulo half of the pipeline parallel scale is determined as the output node number of the batch number; the computing node corresponding to the node number and the output node number is determined as the computing node that performs the output calculation of the micro-batch and the subsequent output reverse calculation. The starting condition for the computing node corresponding to each output node number to perform the output calculation of the micro-batch and the subsequent output reverse calculation is: the remainder obtained by taking the batch number modulo half of the pipeline parallel scale is 0.

[0107] Among them, since in the related art, the output calculation and output reverse calculation of each micro-batch are calculated by the last computing node, the computing node adjustment of the output calculation of the micro-batch and the subsequent output reverse calculation can be performed in the following way:

[0108] Adjust the micro-batches where (num_of_microbatch)%(pipeline_parallel_size / 2)≠0; and the output node number of the computing node that performs the output calculation and output reverse calculation of the micro-batch = pipeline_parallel_size / / 2+(Num_of_microbatch%(pipeline_parallel_size / 2)), and the execution starts when (num_of_microbatch)%(pipeline_parallel_size / 2)==0.

[0109] The following beneficial effects can be obtained by using the above-mentioned load control method based on staggered pipeline parallelism:

[0110] Improve computing resource utilization: By evenly distributing additional computing tasks such as data preprocessing, embedding calculations, output calculations, and loss function calculations to computing nodes in all pipeline stages, the utilization of each computing node is significantly improved. This balanced distribution reduces the idle time of some computing nodes, thereby improving the work efficiency of the entire system.

[0111] Reduced training time: Due to the optimization of load distribution at each stage, the waiting time for data processing and computing tasks is reduced, thus shortening the overall training time of the model. This reduction is particularly significant when training large and complex models, which can accelerate the model iteration and development process.

[0112] Enhance the scalability and flexibility of the system: Through the dynamic load adjustment mechanism, the system can flexibly adjust resource allocation according to real-time computing needs and environmental changes. This is not only applicable to static training tasks, but also effectively cope with the needs of dynamically changing online learning and real-time model updates.

[0113] Reduce energy consumption and operating costs: More efficient resource utilization directly affects energy consumption. By reducing ineffective calculations and waiting time, the overall energy consumption of the system can be reduced, thereby reducing operating costs. This is especially important for training tasks running in large-scale data centers, which can significantly reduce long-term operating expenses.

[0114] Improve model performance and stability: Even load distribution can improve the stability of model training and avoid performance bottlenecks caused by overloading some computing nodes. Such stability is crucial to achieving optimal training results and helps avoid gradient vanishing or exploding problems during training.

[0115] In summary, through sophisticated load balancing design, not only the efficiency of distributed training is improved, but also economic and environmental sustainability benefits are brought, and it has broad application prospects in both commercial and academic research.

[0116] The embodiment of the present disclosure further provides a load control device based on staggered pipeline parallelism, which is applied to distributed training of a target model, and the device includes:

[0117] A parameter determination module, used to determine the allocation parameters required for resource allocation during the training of the target model, wherein the allocation parameters include the number of computing nodes participating in the training of the target model, the pipeline parallel scale, and the number of virtual stages to be executed in each computing node of the target model;

[0118] A computing strategy determination module, used to determine the computing strategy of each computing node for each micro-batch according to the allocation parameters and the node number of each computing node, wherein the computing strategy includes the type of computing to be performed and the number of preheating computing;

[0119] The computing control module is used to control each computing node to perform the calculation of each micro-batch according to the corresponding computing strategy to perform training for the target model until the training is completed.

[0120] In a possible implementation, the calculation type corresponding to each of the computing nodes includes forward calculation and reverse propagation calculation, and the calculation type corresponding to the 1 / 2 computing nodes with the node numbers ranked in front among the computing nodes also includes word embedding calculation and word embedding reverse calculation; the calculation type corresponding to the 1 / 2 computing nodes with the node numbers ranked in the back among the computing nodes also includes output calculation and output reverse calculation.

[0121] In a possible implementation, the computing strategy determination module includes at least one of the following submodules:

[0122] A preheating calculation submodule, used to determine the number of preheating calculations corresponding to each computing node according to the parallel scale of the pipeline, the node number of each computing node and the number of virtual stages;

[0123] A word embedding calculation submodule, used to determine the computing node for executing the word embedding calculation of each micro-batch according to the batch number of each micro-batch and the parallel scale of the pipeline;

[0124] A word embedding reverse calculation submodule, used to determine the computing node for performing the word embedding reverse calculation of each micro-batch according to the batch number of each micro-batch and the parallel scale of the pipeline;

[0125] The output calculation submodule is used to determine the computing nodes that perform the output calculation of each micro-batch and the computing nodes that perform the output reverse calculation of each micro-batch according to the batch number of each micro-batch and the parallel scale of the pipeline.

[0126] In a possible implementation, determining the number of preheating calculations corresponding to each computing node according to the parallel scale of the pipeline, the node number of each computing node, and the number of the virtual stages includes:

[0127] Subtract the node number of the computing node and 1 from the pipeline parallel scale in sequence, and then multiply the result by 2 to obtain a first result; subtract 1 from the number of virtual stages required to be executed by the computing node, and then multiply the result by the pipeline parallel scale to obtain a second result; and use the sum of the first result and the second result as the number of preheating calculations for the corresponding computing node;

[0128] The node sequence number is the value of the node number minus 1.

[0129] In a possible implementation, the computing control module includes:

[0130] The first stage control submodule is used to determine that the computing node has completed the preheating stage calculation and control the computing node to start one forward and one reverse 1F1B stage calculation when it is determined that the number of forward calculations performed by the computing node reaches the preheating calculation number corresponding to the computing node.

[0131] In a possible implementation, determining a computing node for executing word embedding calculations for each micro-batch according to the batch number of each micro-batch and the parallel scale of the pipeline includes:

[0132] For a micro-batch whose batch number is greater than or equal to 2 times the parallel scale of the pipeline, a first remainder is obtained by taking the difference of the batch number of the micro-batch minus 1 modulo the parallel scale of the pipeline, and a second remainder obtained by taking the first remainder modulo half of the parallel scale of the pipeline plus 1 as the word embedding node number corresponding to the batch number;

[0133] The computing node corresponding to the node number and the word embedding node number is determined as the computing node for performing the word embedding calculation of the corresponding micro-batch.

[0134] In a possible implementation, the computing control module includes:

[0135] A transfer submodule is used to transfer the micro-batch that completes the word embedding calculation to the first computing node when it is determined that the computing node that performs the word embedding calculation of the micro-batch is not the first computing node.

[0136] In a possible implementation, determining a computing node for performing word embedding reverse calculation of each micro-batch according to the batch number of each micro-batch and the parallel scale of the pipeline includes:

[0137] When the remainder obtained by taking the batch number of the micro-batch modulo half of the pipeline parallel scale is 0, it is determined that word embedding reverse calculation of the micro-batch needs to be performed; and the difference after the batch number of the micro-batch is subtracted by 1 is modulo half of the pipeline parallel scale to obtain a third remainder, and the difference obtained by subtracting the third remainder from half of the pipeline parallel scale is used as the word embedding reverse node number corresponding to the batch number; the computing node corresponding to the node number and the word embedding reverse node number is determined as the computing node for performing word embedding reverse calculation of the micro-batch, so that the word embedding reverse calculation of the micro-batch is performed after the corresponding word embedding calculation.

[0138] In a possible implementation, determining a computing node for executing output calculation of each micro-batch and a computing node for executing output reverse calculation of each micro-batch according to the batch number of each micro-batch and the parallel scale of the pipeline includes:

[0139] When the remainder obtained by taking the batch number of the micro-batch modulo half of the pipeline parallel scale is not 0, the sum of the quotient obtained by dividing the pipeline parallel scale by 2 and the remainder obtained by taking the batch number modulo half of the pipeline parallel scale is determined as the output node number of the batch number;

[0140] Determine a computing node corresponding to a node number and the output node number as a computing node that performs output calculation of the micro-batch and subsequent output reverse calculation;

[0141] The starting condition for the computing node corresponding to each output node number to execute the output calculation of the micro-batch and the subsequent output reverse calculation is that the remainder obtained by taking the batch number modulo half of the parallel scale of the pipeline is 0.

[0142] It should be noted that, although the above embodiments are used as examples to introduce the load control method and device based on staggered pipeline parallelism, those skilled in the art will understand that the present disclosure should not be limited thereto. In fact, users can flexibly set each step and module according to personal preferences and / or actual application scenarios, as long as they comply with the technical solution of the present disclosure.

[0143] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0144] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0145] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0146] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0147] Figure 3 1 is a block diagram of an apparatus 1900 for load regulation based on staggered pipeline parallelism according to an exemplary embodiment. For example, the apparatus 1900 may be provided as a server or a terminal device. Figure 3 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0148] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0149] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.

[0150] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0151] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0152] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0153] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0154] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0155] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0156] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0157] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0158] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A load control method based on staggered pipeline parallelism, characterized in that: Applied to distributed training of a target model, the method includes: Determine allocation parameters required for resource allocation during training of the target model, wherein the allocation parameters include the number of computing nodes participating in the training of the target model, the pipeline parallel scale, and the number of virtual stages to be executed in each computing node of the target model; Determining a computing strategy of each computing node for each micro-batch according to the allocation parameters and the node number of each computing node, wherein the computing strategy includes a computing type to be performed and a number of preheating calculations; Control each of the computing nodes to perform calculations on each of the micro-batches according to a corresponding computing strategy to perform training for the target model until the training is completed.

2. The method according to claim 1, characterized in that The calculation types corresponding to each of the computing nodes include forward calculation and reverse propagation calculation. The calculation types corresponding to the 1 / 2 computing nodes with the node numbers ranked in front among the computing nodes also include word embedding calculation and word embedding reverse calculation. The calculation types corresponding to the 1 / 2 computing nodes with the node numbers ranked in the back among the computing nodes also include output calculation and output reverse calculation.

3. The method according to claim 2, characterized in that Determining a computing strategy of each computing node for each micro-batch according to the allocation parameter and the node number of each computing node includes at least one of the following operations: Determine the number of preheating calculations corresponding to each computing node according to the parallel scale of the pipeline, the node number of each computing node, and the number of virtual stages; Determining, according to the batch number of each micro-batch and the parallel scale of the pipeline, a computing node for executing word embedding calculations of each micro-batch; Determining, according to the batch number of each micro-batch and the parallel scale of the pipeline, a computing node for performing word embedding reverse calculation of each micro-batch; According to the batch number of each micro-batch and the parallel scale of the pipeline, a computing node for executing output calculation of each micro-batch and a computing node for executing output reverse calculation of each micro-batch are determined.

4. The method according to claim 3, characterized in that Determining the number of preheating calculations corresponding to each computing node according to the parallel scale of the pipeline, the node number of each computing node, and the number of virtual stages, includes: Subtract the node number of the computing node and 1 from the pipeline parallel scale in sequence, and then multiply the result by 2 to obtain a first result; subtract 1 from the number of virtual stages required to be executed by the computing node, and then multiply the result by the pipeline parallel scale to obtain a second result; and use the sum of the first result and the second result as the number of preheating calculations for the corresponding computing node; The node sequence number is the value of the node number minus 1.

5. The method according to claim 4, characterized in that Controlling each of the computing nodes to perform the calculation of each of the micro-batches according to the corresponding computing strategy, including: When it is determined that the number of forward calculations performed by the computing node reaches the number of preheating calculations corresponding to the computing node, it is determined that the computing node completes the preheating stage calculation, and the computing node is controlled to start one forward and one reverse 1F1B stage calculations.

6. The method according to claim 3, characterized in that Determining, according to the batch number of each micro-batch and the parallel scale of the pipeline, a computing node for executing word embedding calculations of each micro-batch, including: For a micro-batch whose batch number is greater than or equal to 2 times the parallel scale of the pipeline, a first remainder is obtained by taking the difference of the batch number of the micro-batch minus 1 modulo the parallel scale of the pipeline, and a second remainder obtained by taking the first remainder modulo half of the parallel scale of the pipeline plus 1 as the word embedding node number corresponding to the batch number; The computing node corresponding to the node number and the word embedding node number is determined as the computing node for performing the word embedding calculation of the corresponding micro-batch.

7. The method according to claim 6, characterized in that Controlling each of the computing nodes to perform the calculation of each of the micro-batches according to the corresponding computing strategy, including: In a case where it is determined that the computing node that performs the word embedding calculation of the micro-batch is not the first computing node, the micro-batch that completes the word embedding calculation is transferred to the first computing node.

8. The method according to claim 3, characterized in that Determining, according to the batch number of each micro-batch and the parallel scale of the pipeline, a computing node for performing word embedding reverse calculation of each micro-batch, including: When the remainder obtained by taking the batch number of the micro-batch modulo half of the pipeline parallel scale is 0, it is determined that word embedding reverse calculation of the micro-batch needs to be performed; and the difference after the batch number of the micro-batch is subtracted by 1 is modulo half of the pipeline parallel scale to obtain a third remainder, and the difference obtained by subtracting the third remainder from half of the pipeline parallel scale is used as the word embedding reverse node number corresponding to the batch number; the computing node corresponding to the node number and the word embedding reverse node number is determined as the computing node for performing word embedding reverse calculation of the micro-batch, so that the word embedding reverse calculation of the micro-batch is performed after the corresponding word embedding calculation.

9. The method according to claim 3, characterized in that: Determining, according to the batch number of each micro-batch and the parallel scale of the pipeline, a computing node for executing output calculation of each micro-batch and a computing node for executing output reverse calculation of each micro-batch, including: When the remainder obtained by taking the batch number of the micro-batch modulo half of the pipeline parallel scale is not 0, the sum of the quotient obtained by dividing the pipeline parallel scale by 2 and the remainder obtained by taking the batch number modulo half of the pipeline parallel scale is determined as the output node number of the batch number; Determine a computing node corresponding to a node number and the output node number as a computing node that performs output calculation of the micro-batch and subsequent output reverse calculation; The starting condition for the computing node corresponding to each output node number to execute the output calculation of the micro-batch and the subsequent output reverse calculation is that the remainder obtained by taking the batch number modulo half of the parallel scale of the pipeline is 0.

10. A load control device based on staggered pipeline parallel operation, characterized in that: Applied to distributed training for a target model, the device comprises: A parameter determination module, used to determine the allocation parameters required for resource allocation during the training of the target model, wherein the allocation parameters include the number of computing nodes participating in the training of the target model, the pipeline parallel scale, and the number of virtual stages to be executed in each computing node of the target model; A computing strategy determination module, used to determine the computing strategy of each computing node for each micro-batch according to the allocation parameters and the node number of each computing node, wherein the computing strategy includes the type of computing to be performed and the number of preheating computing; The computing control module is used to control each computing node to perform the calculation of each micro-batch according to the corresponding computing strategy to perform training for the target model until the training is completed.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 9 when executing the instructions stored in the memory.

12. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images