Interleaved pipeline scheduling method and apparatus, device, and storage medium

Abstract

The present disclosure provides an interleaved pipeline scheduling method and apparatus, a device, and a storage medium, relating to the technical field of computers, and in particular to the technical field of artificial intelligence such as deep learning, neural networks, and parallel computing. The specific implementation scheme comprises: on the basis of a pipeline segmentation dimension, an interleaving dimension, and an accumulation count, obtaining micro-batch scheduling parameters; on the basis of the micro-batch scheduling parameters, determining a transmission interval for micro-batches requiring buffered scheduling; and on the basis of the transmission interval and a buffer unit, performing buffered scheduling on computation results of micro-batches requiring delayed transmission. According to the present disclosure, a buffer unit, such as a buffer queue, can be created to buffer computation results from some computation units in the buffer unit, and to delay the transmission of the computation results. In this way, the transmission of computation results for some micro-batches can be delayed, thereby improving the applicability of interleaved pipeline scheduling.

Description

Interleaved pipeline scheduling method, device, equipment and storage medium Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to artificial intelligence technologies such as deep learning, neural networks, and parallel computing. Background Art

[0002] In the field of deep learning, distributed strategies can be used to improve the computational efficiency of computing units. Pipeline parallelism is a commonly used distributed strategy. Interleaved pipelines can further optimize the bubble occupancy rate of conventional pipelines.

[0003] Summary of the Invention

[0004] The present disclosure provides a staggered pipeline scheduling method, apparatus, device, and storage medium.

[0005] According to one aspect of the present disclosure, a staggered pipeline scheduling method is provided, comprising:

[0006] According to the pipeline segmentation dimension, interleaving dimension and accumulation times, the micro-batch scheduling parameters are obtained;

[0007] Determine the sending interval of the micro-batch that needs to be cached and scheduled according to the micro-batch scheduling parameters;

[0008] According to the sending interval and the cache unit, cache scheduling is performed on the calculation results of the micro-batch that needs to be delayed in sending.

[0009] According to another aspect of the present disclosure, there is provided a staggered pipeline scheduling device, comprising:

[0010] The parameter acquisition module is used to obtain the micro-batch scheduling parameters based on the pipeline segmentation dimension, interleaving dimension, and accumulation number;

[0011] An interval determination module, configured to determine a sending interval of a micro-batch that needs to be cached and scheduled according to the micro-batch scheduling parameters;

[0012] The cache scheduling module is used to perform cache scheduling on the calculation results of the micro-batch that needs to be delayed in sending according to the sending interval and the cache unit.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any method in the embodiments of the present disclosure.

[0017] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements any one of the methods according to the embodiments of the present disclosure.

[0019] In the disclosed embodiments, a cache mechanism is introduced to assist in the scheduling of the staggered pipeline, and the sending interval of the micro-batches that need to be cached is determined by the micro-batch scheduling parameters. This can delay the sending of the calculation results of some micro-batches, and does not limit the number of micro-batches scheduled at one time, thereby improving the applicability of the staggered pipeline scheduling.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] FIG1 is a schematic flow chart of a staggered pipeline scheduling method according to an embodiment of the present disclosure.

[0023] FIG2 is a schematic diagram of a pipeline scheduling scenario according to the present disclosure.

[0024] FIG3 is a schematic diagram of a staggered pipeline scheduling scenario according to the present disclosure.

[0025] FIG4 is a flow chart of a staggered pipeline scheduling method according to another embodiment of the present disclosure.

[0026] FIG5 is a schematic flow chart of a staggered pipeline scheduling method according to another embodiment of the present disclosure.

[0027] FIG6 is a flowchart of a staggered pipeline scheduling method according to another embodiment of the present disclosure.

[0028] FIG7 is a schematic diagram of a scenario of staggered pipeline cache scheduling according to the present disclosure.

[0029] FIG8 is a schematic diagram of a scenario of staggered pipeline cache scheduling according to the present disclosure.

[0030] FIG9 is a schematic flow chart of a staggered pipeline scheduling method according to another embodiment of the present disclosure.

[0031] FIG10 is a flow chart of interleaved pipeline cache scheduling for backward computation.

[0032] FIG11 is a flow chart of interleaved pipeline cache scheduling for forward computation.

[0033] FIG12 is a schematic structural diagram of an interleaved pipeline scheduling device according to an embodiment of the present disclosure.

[0034] FIG13 is a schematic structural diagram of an interleaved pipeline scheduling device according to another embodiment of the present disclosure.

[0035] FIG14 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0037] FIG1 is a flow chart of a staggered pipeline scheduling method according to an embodiment of the present disclosure, the method comprising:

[0038] S101. Obtain micro-batch scheduling parameters based on the pipeline segmentation dimension, interleaving dimension, and accumulation times.

[0039] S102: Determine the sending interval of the micro-batch that needs to be cached and scheduled according to the micro-batch scheduling parameters;

[0040] S103 : Cache scheduling is performed on the calculation results of the micro-batch that needs to be delayed in sending according to the sending interval and the cache unit.

[0041] In the disclosed embodiment, since artificial intelligence models such as neural networks require large computing power during training, they can be processed in parallel by multiple computing units. Different computing units can be used to perform calculations at different stages of the model to achieve pipeline parallel computing. In pipeline parallel computing, data can be transmitted between adjacent computing units through a communication link. An example of a pipeline scheduling scenario is shown in Figure 2. Among them, the horizontal direction represents the number of calculation steps of a computing unit. In this pipeline scheduling scenario, a computing unit can calculate a forward stage and a backward stage of the model. The rectangle corresponds to the number of steps of the forward calculation, and the square corresponds to the number of steps of the backward calculation. Assuming that 8 micro-batches need to be calculated, they can be divided into two blocks. First, four computing units are used to calculate the micro-batches of the first block (serial numbers 1 to 4) in sequence, and then the micro-batches of the second block (serial numbers 5 to 8) are calculated in sequence.

[0042] In the embodiment of the present disclosure, each computing unit of the staggered pipeline scheduling can perform computing functions of multiple stages of the model training process. An example of a staggered pipeline scheduling scenario is shown in Figure 3. The number of staggers in the staggered pipeline scheduling scenario is 2. A computing unit can calculate multiple forward stages and multiple backward stages of the model. After the first computing unit to the fourth computing unit perform forward calculations on a micro-batch in turn, the fourth computing unit needs to send the calculation results back to the first computing unit for forward calculation again. For example, after the first computing unit performs the first forward calculation on micro-batch 1 in step 1, it performs the second forward calculation on micro-batch 1 in step 5. The backward is similar. The first computing unit needs to send the backward calculation results back to the fourth computing unit for backward calculation again. Other details will not be repeated.

[0043] Because each computation unit directly passes its results to the next computation unit after computation, staggered pipeline parallelism often requires limiting the batch size. For example, the batch size must be an integer multiple of the pipeline partitioning dimension. Many batch sizes cannot meet this requirement, limiting the use cases of staggered pipeline scheduling.

[0044] In the embodiment of the present disclosure, a cache unit such as a cache queue can be created to cache the calculation results of some computing units in the cache unit and delay the sending of the calculation results. This can remove the restrictions on batch size and flexibly support pipeline scheduling of different batch sizes. For example, the forward calculation result of the last computing unit can be cached in the cache unit, and then the calculation result can be transmitted to the first computing unit after a certain number of steps. For another example, the backward calculation result of the first computing unit can be cached in the cache unit, and then the calculation result can be transmitted to the last computing unit after a certain number of steps.

[0045] In an embodiment of the present disclosure, in order to delay the transmission of calculation results in a cache unit, micro-batch scheduling parameters can be calculated based on the pipeline partitioning dimension, the interleaving dimension, and the number of accumulations. The pipeline partitioning dimension can be determined based on the number of computing units used. For example, if the number of computing units is four, the pipeline partitioning dimension is also four. The sample set of the model to be trained may include multiple micro-batches, each of which may include one or more samples. The number of accumulations can be determined based on the number of micro-batches that can be processed in a single interleaved pipeline scheduling. For example, if the number of micro-batches that can be processed in a single interleaved pipeline scheduling is 9, the number of accumulations is also 9. If the number of micro-batches that can be processed in a single interleaved pipeline scheduling is 10, the number of accumulations is also 10. If the number of micro-batches that can be processed in a single interleaved pipeline scheduling does not meet an integer multiple of the pipeline partitioning dimension, a portion of the data that needs to be transmitted via the communication link can be cached. The interleaving dimension may include the number of stages of the model training process that a computing unit needs to interleave for a micro-batch. For example, a computing unit performs one stage of forward computation on a micro-batch in step 1 and another stage of forward computation on the same micro-batch in step 6. In this case, the interleaving dimension can be 2. The micro-batch scheduling parameters can be used to determine the sending interval of the micro-batches that need to be cached in an interleaved pipeline scheduling. The sending interval can be used to delay the sending of data stored in the cache unit (the computation results of the micro-batch).

[0046] According to the embodiments of the present disclosure, a cache mechanism is introduced to assist in the scheduling of the staggered pipeline, and the sending interval of the micro-batches that need to be cached is determined by the micro-batch scheduling parameters. This can delay the sending of the calculation results of some micro-batches, and does not limit the number of micro-batches scheduled at one time, thereby improving the applicability of the staggered pipeline scheduling.

[0047] In one embodiment, the pipeline splitting dimension includes the number of computing units used to perform pipeline scheduling on the model training process.

[0048] In the disclosed embodiments, the size of the pipeline split dimension can be determined based on the number of computing units that are pipeline-scheduled during the model training process. For example, a computing unit may include one or more graphics processing units (GPUs), and a GPU may also be divided into multiple computing units (or a computing unit may include some of the computing functions of a GPU). The size of the pipeline split dimension can be determined based on the number of available computing units.

[0049] According to the embodiments of the present disclosure, the size of the pipeline segmentation dimension can be determined in combination with the actual number of computing units, which is conducive to the rational use of computing units and improves the computing power of staggered pipeline scheduling.

[0050] In one embodiment, the model training process is divided into multiple stages, and the interleaving dimension includes the number of stages that a computing unit needs to interleave for a micro-batch.

[0051] In the disclosed embodiment, the training process of the model can be divided into multiple stages, and the specific division method can be determined with reference to the structure, function, etc. of the model. The size of the interleaving dimension is related to the number of computing units and the number of stages into which the training process of the model is divided. For example, the training process of the model is divided into 8 stages, the number of computing units is 4, and the size of the interleaving dimension can be 2. In this case, one computing unit needs to interleave the calculation of 2 stages of the model for one micro-batch. The first computing unit performs the calculation of stage 1 and stage 5; the second computing unit performs the calculation of stage 2 and stage 6; the first computing unit performs the calculation of stage 3 and stage 7; the first computing unit performs the calculation of stage 4 and stage 8.

[0052] According to the embodiment of the present disclosure, the size of the interleaving dimension of the pipeline can be determined in combination with the number of stages into which the model training process is divided, which is conducive to improving the computational efficiency of interleaved pipeline scheduling.

[0053] Figure 4 is a flow chart of an interleaved pipeline scheduling method according to another embodiment of the present disclosure, which may include one or more features of the above-mentioned interleaved pipeline scheduling method. In one embodiment, S101 obtains micro-batch scheduling parameters based on the pipeline segmentation dimension, interleaving dimension, and accumulation number, including:

[0054] S401: Obtain the number of micro-batches included in the first block according to the pipeline segmentation dimension and the cumulative number of times;

[0055] S402: Obtain a total number of steps of the first block according to the number of micro-batches included in the first block and the interleaving dimension;

[0056] S403: Obtain the number of jump steps according to the pipeline segmentation dimension and the cumulative number of times.

[0057] In the disclosed embodiment, the total number of micro-batches included in an interleaved pipeline scheduling may not be an integer multiple of the pipeline segmentation dimension. In this case, the micro-batches included in an interleaved pipeline scheduling are divided into blocks. For example, the number of micro-batches included in a block (which may be referred to as a first block) that is not an integer multiple of the pipeline segmentation dimension can be calculated based on the pipeline segmentation dimension and the cumulative number of times. For example, the number of micro-batches included in the first block can be obtained by adding the sum of the segmentation dimensions after the cumulative number of times to find the remainder of the segmentation dimension. In addition to the micro-batches included in the first block, the remaining micro-batches in the micro-batches included in an interleaved pipeline scheduling can send the calculation results without using a cache delay, but directly send the calculation results. For example, for an interleaved pipeline scheduling, the split dimension is 4 and the cumulative number is 9. The number of micro-batches included in the first block is (9%4)+4=5. Then, micro-batches 1, 2, 3, 4, and 5 in the first block need to be cached, while the remaining micro-batches 6, 7, 8, and 9 are scheduled normally.

[0058] In the embodiment of the present disclosure, the total number of steps that the first block needs to execute in a computing unit can be obtained based on the number of micro-batches and the interleaving dimension included in the first block. The total number of steps may include the total number of steps of the forward calculation and / or the total number of steps of the backward calculation. For example, the product of the number of micro-batches included in the first block and the interleaving dimension can be used as the total number of steps of the first block. If the number of micro-batches included in the first block is 5 and the interleaving dimension is 2, the total number of steps of the first block is 5*2=10, and the total number of forward calculations performed by each computing unit on all micro-batches in the first block is 10, and the total number of backward calculations is also 10.

[0059] In the disclosed embodiment, the number of jump steps can be used to determine the number of steps required from storing the calculation result of a micro-batch of the first block in the cache queue to delaying the sending of the result. The remainder obtained by taking the remainder of the pipeline partitioning dimension using the cumulative number of times can be used as the number of jump steps. For example, if the partitioning dimension is 4 and the cumulative number of times is 9, the number of jump steps is 9%4=1. In this case, the calculation result of a micro-batch of the first block requires 1 step from being cached to being sent.

[0060] According to an embodiment of the present disclosure, by calculating micro-batch scheduling parameters such as the number of micro-batches included in the first block, the total number of steps in the first block, and the number of jump steps, the calculation results of the micro-batches included in the first block in the cache queue can be delayed.

[0061] FIG5 is a flow chart of an interleaved pipeline scheduling method according to another embodiment of the present disclosure, which method may include one or more features of the above-mentioned interleaved pipeline scheduling method. In one embodiment, S102 determines the sending interval of the micro-batch that needs to be cached and scheduled based on the micro-batch scheduling parameters, including:

[0062] S501: Obtain a first boundary of the sending interval according to the jump step number, where the first boundary is used to determine a starting step number for sending buffered data.

[0063] S502: Obtain a second boundary of the sending interval according to the first boundary, the number of micro-batches included in the first block, and the interleaving dimension, where the second boundary is used to determine the number of steps at which sending of the cached data needs to be stopped.

[0064] In the disclosed embodiment, the sending interval can be understood as the interval of the number of steps required to send the data in the cache unit to other computing units. The sending interval can have a first boundary and a second boundary. The first boundary can also be called the left boundary, which can be used to determine the number of steps required to start sending the calculation results of the micro-batch in the cache queue. The second boundary can also be called the right boundary, which can be used to determine the number of steps required to start sending the calculation results of the micro-batch in the cache queue. The sending interval can be a closed interval, a half-open and half-closed interval, or an open interval. For example, the number of jump steps can be used as the first boundary of the sending interval, and the sending interval does not include the first boundary itself. For another example, the number of jump steps plus 1 can also be used as the first boundary of the sending interval, and the sending interval includes the first boundary itself.

[0065] In the embodiment of the present disclosure, the second boundary of the sending interval can be calculated based on the first boundary, the number of micro-batches included in the first block, and the interleaving dimension. For example, the difference between the interleaving dimension minus 1 is multiplied by the number of micro-batches included in the first block, and the resulting product is added to the first boundary to obtain the second boundary. For example, if the first boundary is 1, the number of micro-batches included in the first block is 5, and the interleaving dimension is 2, the second boundary is calculated to be 1+5*(2-1)=6.

[0066] In the disclosed embodiment, the relationship between the sending interval and the buffer interval can be determined based on the number of jump steps. The buffer interval can include the number of steps required to store the calculation results of the micro-batch into the cache queue. The difference between the boundaries of the sending interval and the buffer interval can be equal to the number of jump steps or calculated based on the number of jump steps. For example, the buffer interval is [1, 6) and the sending interval is [2, 7). For another example, the buffer interval is [1, 6) and the sending interval is (1, 6).

[0067] According to an embodiment of the present disclosure, the sending of data in the cache queue can be controlled by the boundary of the sending interval, so that the calculation results of each micro-batch in the first block can be delayed and sent to the corresponding computing unit.

[0068] FIG6 is a flow chart of an interleaved pipeline scheduling method according to another embodiment of the present disclosure, which method may include one or more features of the above-mentioned interleaved pipeline scheduling method. In one embodiment, the cache unit may include a cache queue, such as a forward cache queue. According to the sending interval and the cache unit, the calculation results of the micro-batch that needs to be delayed in sending are cached and scheduled, including: according to the current number of steps of the micro-batch being processed by the last computing unit, the sending interval and the forward cache queue, the calculation results of the micro-batch that needs to be delayed in the forward calculation process of the last computing unit are cached and scheduled.

[0069] In the disclosed embodiment, the forward cache queue (or referred to as the forward calculation cache queue, the forward propagation cache queue, etc.) can be used to store the calculation results of the micro-batch in the forward calculation process, such as tensors. In the forward calculation process, it is necessary to cache the calculation result of the last computing unit first, and then delay it to be sent to the first computing unit after the jump step number. In this process, the calculation result of the micro-batch of the last computing unit can be first stored in the forward cache queue according to the current step number, and then the current step number and the sending interval are compared, so that the calculation result stored in the forward cache queue is sent to the first computing unit after the interval jump step number.

[0070] According to the embodiments of the present disclosure, staggered pipeline scheduling can be performed on the training sample set during the model training process through cache queues and sending intervals. There is no need to limit the relationship between the number of micro-batches and the segmentation dimension of the training sample set, which improves the applicability of staggered pipeline scheduling.

[0071] In one embodiment, cache scheduling is performed on the calculation results of the micro-batch that needs to be delayed in the forward calculation process of the last computing unit according to the current step number of the micro-batch being processed by the last computing unit, the sending interval, and the forward cache queue, including:

[0072] S601: If the current step number of the micro-batch being processed by the last computing unit is less than the total step number of the first block, and the current step number is not the last part of the interleaved step number, store the tensor corresponding to the current step number in the forward cache queue;

[0073] S602: Determine whether to send the head tensor in the forward buffer queue to the first computing unit according to the current step number and the sending interval.

[0074] In the disclosed embodiment, for forward computation, if the number of computation steps currently being executed by the last computation unit (i.e., the current step number) is less than the total number of steps in the first block, this indicates that the current step number is the number of computation steps for the micro-batch within the first block. If the current step number is greater than or equal to the total number of steps in the first block, this indicates that the current step number is the number of computation steps for micro-batches prior to the first block. In this case, there is no need to cache the computation results for the micro-batch.

[0075] If the current step number is less than the total number of steps in the first block, it can be further determined whether the current step number is the last part of the interleaved step number. The last part of the interleaved step number is related to the interleaving dimension VPP. For example, the difference between VPP minus 1 and the number of micro-batches included in the first block is multiplied by the product as the first threshold, that is, (VPP-1)*the number of micro-batches included in the first block. If the current step number is greater than the first threshold, it means that the current step number is the last part of the interleaved step number. If the current step number is not the last part of the interleaved step number, the tensor of the step number is stored in the forward cache queue. Referring to Figure 7, the current step number of the last computing unit can be accumulated starting from 1 (or 0), that is, the current step number of the first step is 1 and the current step number of the second step is 2. If the number of micro-batches included in the first block is 5 and the interleaving dimension is 2, then the first threshold is 5. If the current step number of the last computing unit is 1 to 5, it is not the last part of the interleaved step number, and if the current step number is 6 to 10, it is the last part of the interleaved step number. 8 , if the number of micro-batches included in the first block is 5 and the interleaving dimension is 3, the first threshold is 10. The current step number of the last computation unit from 1 to 11 is not the last part of the interleaved step number, and the current step number from 11 to 15 is the last part of the interleaved step number.

[0076] In the disclosed embodiment, the number of steps from the beginning of the first segment to the number of steps corresponding to the first threshold value can be understood as a buffer interval. The buffer interval is different from the sending interval and can delay the data in the sending buffer queue. For example, the current number of steps that the last computing unit calculates micro-batch 1 is 1, and the current number of steps is less than the first threshold value, so the result of calculation of micro-batch 1 can be stored in the first item (head) of the forward buffer queue. However, the current number of steps is not in the sending interval, so the data in the forward buffer queue are not sent. The current number of steps that the last computing unit calculates micro-batch 2 is 2, and the current number of steps is less than the first threshold value, so the result of calculation of micro-batch 2 can be stored in the second item of the forward buffer queue. And the current number of steps is in the sending interval, so the header data in the forward buffer queue can be sent to the first computing unit, and the second item of the forward buffer queue can be changed into the first item. In another way, the header data in the forward buffer queue can also be sent to the first computing unit first, and then the result of calculation of micro-batch 2 can be cached.

[0077] According to an embodiment of the present disclosure, after the tensors of the micro-batch calculated by the last computing unit are stored in the forward cache queue, it is possible to determine whether the current step number is to send the head tensor in the forward cache queue to the first computing unit based on the current step number and the sending interval. The sending of the cached tensor can be delayed, thereby increasing the number of micro-batches applicable to the interleaved pipeline scheduling.

[0078] In one embodiment, determining whether to send the head tensor in the forward buffer queue to the first computing unit according to the current step number and the sending interval includes:

[0079] If the current step number is within the sending interval, the scheduling value of the head tensor in the forward cache queue is set to the first value; or if the current step number is not within the sending interval, the scheduling value of the head tensor in the forward cache queue is set to the second value;

[0080] The header tensor whose scheduling value is the first value is sent to the first computing unit.

[0081] In the disclosed embodiment, the first value and the second value are different. For example, the first value can be a non-null value, and the second value can be a null value (none). When the current step number is within the sending interval, the scheduling value of the head tensor in the forward cache can be set to a non-null value, such as 1; when the current step number is not within the sending interval, the scheduling value of the head tensor in the forward cache queue is set to a null value, such as none. For example, the number of micro-batches included in the first block is 5, the interleaving dimension is 2, the current step number starts to accumulate from 1, and the sending interval is (1,6]. Then, the tensor of micro-batch 1 calculated by the current step number 1 can be first stored in the head of the forward cache queue. Then, since the current step number 1 is not in the sending interval (1,6], the scheduling value of the head tensor in the forward cache queue is set to a null value. Since the scheduling value of the head tensor is a null value, the current step number 1 does not send the head tensor. Next, the current step accumulation becomes 2, and the current step number 2 is in the sending interval (1,6], and the scheduling value of the head tensor in the forward cache queue is set to 1. Since the scheduling value of the head tensor is 1, the current step number 2 sends the head tensor (the tensor of micro-batch 1). After sending, the tensor of micro-batch 1 can be deleted, and the tensor of micro-batch 2 can be changed to the head tensor in the forward cache queue.

[0082] According to an embodiment of the present disclosure, the scheduling value of the head tensor in the forward cache queue can be used to control the sending timing of the last computing unit to the first computing unit, thereby delaying the sending of a portion of micro-batch tensors and increasing the number of micro-batches applicable to interleaved pipeline scheduling.

[0083] In one embodiment, cache scheduling is performed on the calculation results of the micro-batch that needs to be delayed in the forward calculation process of the last computing unit according to the current step number of the micro-batch being processed by the last computing unit, the sending interval, and the forward cache queue, including:

[0084] When the current step number is not less than the total step number of the first block, the scheduling value of the tensor corresponding to the current step number is set to the second value.

[0085] In the embodiment of the present disclosure, when the current number of steps is not less than the total number of steps of the first block, the calculation performed corresponds to other blocks that do not require cache scheduling. The scheduling value of the tensor corresponding to the current number of steps can be set to a second value, such as an empty value. The tensor corresponding to the current number of steps does not need to be stored in the forward cache queue, nor does it need to be delayed in sending.

[0086] According to the embodiment of the present disclosure, other blocks that do not require cache scheduling can be sent normally, which can reduce scheduling complexity and improve calculation efficiency.

[0087] FIG9 is a flow chart of an interleaved pipeline scheduling method according to another embodiment of the present disclosure, which may include one or more features of the interleaved pipeline scheduling method described above. In one embodiment, the cache unit may include a backward cache queue, and cache scheduling is performed for the computation results of the micro-batch that needs to be delayed based on the sending interval and the cache unit. The method also includes: caching the computation results of the micro-batch that needs to be delayed during the backward computation of the first computing unit based on the current step number of the micro-batch being processed by the first computing unit, the sending interval, and the backward cache queue.

[0088] In the disclosed embodiment, the backward cache queue (or referred to as the backward calculation cache queue, the backward propagation cache queue, etc.) can be used to store the calculation results of the micro-batch in the backward calculation process, such as tensors. In the backward calculation process, it is necessary to cache the calculation results of the first computing unit first, and then delay sending to the last computing unit after the jump step number. In this process, the calculation results of the micro-batch of the first computing unit can be first stored in the backward cache queue according to the current step number, and then the current step number and the sending interval are compared to send the calculation results stored in the backward cache queue to the last computing unit after the interval jump step number.

[0089] According to the embodiments of the present disclosure, staggered pipeline scheduling can be performed on the training sample set during the model training process through cache queues and sending intervals. There is no need to limit the relationship between the number of micro-batches and the segmentation dimension of the training sample set, which improves the applicability of staggered pipeline scheduling.

[0090] In one embodiment, cache scheduling is performed for calculation results of the micro-batch that needs to be delayed in the backward calculation process of the first computing unit according to the current step number of the micro-batch being processed by the first computing unit, the sending interval, and the backward cache queue, including:

[0091] S901: When the current step number of the micro-batch being processed by the first computing unit is less than the total step number of the first block, and the current step number is not the last part of the interleaved step number, store the tensor corresponding to the current step number in the backward cache queue;

[0092] S902: Determine whether to send the head tensor in the backward buffer queue to the last computing unit according to the current step number and the sending interval.

[0093] In the disclosed embodiment, for backward computation, if the number of computation steps currently executed by the first computation unit (i.e., the current step number) is less than the total number of steps in the first block, this indicates that the current step number is the number of computation steps for the micro-batch within the first block. If the current step number is greater than or equal to the total number of steps in the first block, this indicates that the current step number is the number of computation steps for micro-batches prior to the first block. In this case, there is no need to cache the computation results of the micro-batch.

[0094] For the case where the current step number is less than the total number of steps in the first block, it can be further determined whether the current step number is the last part of the interleaved step number. The last part of the interleaved step number is related to the interleaved dimension VPP. For example, the difference of VPP minus 1 multiplied by the product of the number of micro-batches included in the first block is used as the first threshold. If the current step number is greater than the first threshold, it means that the current step number is the last part of the interleaved step number. In the case where the current step number is not the last part of the interleaved step number, the tensor of the step number is stored in the backward cache queue. Referring to Figure 7, the current step number of the first computing unit can be accumulated from 1 (or 0), that is, the current step number of the first step is 1 and the current step number of the second step is 2. If the number of micro-batches included in the first block is 5 and the interleaved dimension is 2, the first threshold is 5. The current step number of the first computing unit from 1 to 5 is not the last part of the interleaved step number, and the current step number from 6 to 10 is the last part of the interleaved step number. 8 , if the number of micro-batches included in the first block is 5 and the interleaving dimension is 3, the first threshold is 10. The current step number of the first computation unit from 1 to 11 is not the last part of the interleaved step number, and the current step number from 11 to 15 is the last part of the interleaved step number.

[0095] In the disclosed embodiment, the number of steps from the beginning of the first segment to the number of steps corresponding to the first threshold value can be understood as a buffer interval. The buffer interval is different from the sending interval and can delay the data in the sending buffer queue. For example, the current number of steps that the first computing unit calculates micro-batch 1 is 1, and the current number of steps is less than the first threshold value, so the result of calculation of micro-batch 1 can be stored in the first item (head) of the backward buffer queue. However, the current number of steps is not in the sending interval, so the data in the backward buffer queue are not sent. The current number of steps that the first computing unit calculates micro-batch 2 is 2, and the current number of steps is less than the first threshold value, so the result of calculation of micro-batch 2 can be stored in the second item of the backward buffer queue. And the current number of steps is in the sending interval, so the header data in the backward buffer queue can be sent to last computing unit, and the second item of the backward buffer queue is changed to the first item. In another way, the header data in the backward buffer queue can also be sent to last computing unit first, and then the result of calculation of micro-batch 2 can be cached.

[0096] According to an embodiment of the present disclosure, after the tensors of the micro-batch calculated by the first computing unit are stored in the backward cache queue, it can be determined whether the current step number is to send the head tensor in the backward cache queue to the last computing unit based on the current step number and the sending interval. The sending of the cached tensor can be delayed, thereby increasing the number of micro-batches applicable to the interleaved pipeline scheduling.

[0097] In one embodiment, determining whether to send the head tensor in the backward buffer queue to the last computing unit according to the current step number and the sending interval includes:

[0098] When the current step number is within the sending interval, the scheduling value of the head tensor in the backward buffer queue is set to the first value; or, when the current step number is not within the sending interval, the scheduling value of the head tensor in the backward buffer queue is set to the second value;

[0099] The header tensor whose scheduling value is the first value is sent to the last computing unit.

[0100] In the disclosed embodiment, the first value and the second value are different. For example, the first value can be a non-null value, and the second value can be a null value (none). When the current step number is within the sending interval, the scheduling value of the head tensor in the backward buffer can be set to a non-null value, such as 1; when the current step number is not within the sending interval, the scheduling value of the head tensor in the backward buffer queue is set to a null value, such as none. For example, the number of micro-batches included in the first block is 5, the interleaving dimension is 2, the current step number starts to accumulate from 1, and the sending interval is (1,6]. Then, the tensor of micro-batch 1 calculated by the current step number 1 can be first stored in the head of the backward cache queue. Then, since the current step number 1 is not in the sending interval (1,6], the scheduling value of the head tensor in the backward cache queue is set to a null value. Since the scheduling value of the head tensor is a null value, the current step number 1 does not send the head tensor. Next, the current step accumulation becomes 2, and the current step number 2 is in the sending interval (1,6], and the scheduling value of the head tensor in the backward cache queue is set to 1. Since the scheduling value of the head tensor is 1, the current step number 2 sends the head tensor (the tensor of micro-batch 1). After sending, the tensor of micro-batch 1 can be deleted, and the tensor of micro-batch 2 becomes the head tensor in the backward cache queue.

[0101] According to an embodiment of the present disclosure, the sending timing of the first computing unit to the last computing unit can be controlled by the scheduling value of the head tensor in the backward cache queue, thereby delaying the sending of a portion of micro-batches of tensors and increasing the number of micro-batches applicable to the interleaved pipeline scheduling.

[0102] In one embodiment, cache scheduling is performed on calculation results of the micro-batch that need to be delayed in sending during backward calculation by the last computing unit based on the current step number of the micro-batch being processed by the first computing unit, the sending interval, and the forward cache queue, including: when the current step number is not less than the total step number of the first block, setting the scheduling value of the tensor corresponding to the current step number to the second value.

[0103] In the embodiment of the present disclosure, when the current number of steps is not less than the total number of steps of the first block, the calculation performed corresponds to other blocks that do not require cache scheduling. The scheduling value of the tensor corresponding to the current number of steps can be set to a second value, such as an empty value. The tensor corresponding to the current number of steps does not need to be stored in the backward cache queue, nor does it need to be delayed in sending.

[0104] According to the embodiment of the present disclosure, other blocks that do not require cache scheduling can be sent normally, which can reduce scheduling complexity and improve calculation efficiency.

[0105] In one application example, an interleaved pipeline can have no restrictions on the batch size (the number of micro-batches processed at a time), and does not require the batch size to be divisible by the partitioning dimension (pp). This disclosure adjusts the scheduling method of the interleaved pipeline to support scheduling of any batch size. Figure 7 shows a scenario with a batch size of 9 and an interleaving dimension of 2.

[0106] As shown in Figure 7, in this scenario, the pipeline segmentation dimension is num_stages = 4, the interleaving dimension is num_model_chunks = 2, and the number of accumulations is acc_steps = 9.

[0107] First, calculate the following micro-batch scheduling parameters: the number of micro-batches in the first chunk (first_chunk_acc), the total number of steps in the first chunk (first_chunk_steps), and the number of steps to be skipped (skip_steps). The formula example is as follows:

[0108] first_chunk_acc=acc_steps%num_stages+num_stages.

[0109] first_chunk_steps=first_chunk_acc*num_model_chunks.

[0110] skip_steps=acc_steps%num_stages.

[0111] In one example, referring to FIG7 , first_chunk_acc = 9% 4 + 4 = 5; first_chunk_steps = 5*2 = 10; and skip_steps = 9% 4 = 1. In another example, referring to FIG8 , if the interleaving dimension is num_model_chunks = 3 and other parameters are similar to those in FIG7 , first_chunk_acc = 9% 4 + 4 = 5; first_chunk_steps = 5*3 = 15; and skip_steps = 9% 4 = 1. In another example, if the number of accumulations is acc_steps = 10 and other parameters are similar to those in FIG7 , first_chunk_acc = 10% 4 + 4 = 6; first_chunk_steps = 6*2 = 12; and skip_steps = 10% 4 = 2.

[0112] Secondly, construct two buffer queues: the forward buffer queue fwd_buffer_queue and the backward buffer queue bwd_buffer_queue.

[0113] Next, calculate the limits of the first and last computing devices: the left limit is left_id and the right limit is right_id. The calculation formula example is as follows:

[0114] left_id = skip_steps;

[0115] right_id=left_id+first_chunk_acc*(num_model_chunks-1).

[0116] For example, referring to FIG7 , left_id=1, right_id=1+5*(2-1)=6.

[0117] Then, for the first computing unit (pp-stage0), the scheduling process of the modified interleaved pipeline is shown in Figure 10:

[0118] S1001: First, determine whether the current step number step_id of the first computation unit is less than the total number of steps in the first chunk, first_chunk_steps. If so, proceed to S1002 for further determination. If step_id is not less than first_chunk_steps, proceed to step S1005. If step_id is not less than first_chunk_steps, proceed to S1002. If the determination result is yes, proceed to S1005.

[0119] S1002: Determine whether the first computing unit (pp-stage 0) is currently executing the last part of the interleaved schedule (or determine whether it is pp-stage 0 of the interleaved schedule). If not, place the computation result corresponding to the current step, such as a tensor, into bwd_buffer_queue. Continue with S1003. Otherwise, execute S1005.

[0120] S1003. Then check whether step_id is between left_id and right_id (if step_id is numbered from 0, the sending interval may include left_id but not right_id; if step_id is numbered from 1, the sending interval may include right_id but not left_id). If yes, execute S1004; if not, execute S1005.

[0121] S1004: Set a value for the tensor in bwd_buffer_queue. The value is not None. Then execute S1005.

[0122] S1005: Set tensor to None. Then execute S1006.

[0123] S1006. Determine whether tensor is None. If the tensor is not None, send the tensor in the bwd_buffer_queue to the last computing unit pp-last-satge. If the tensor is None, do not send it.

[0124] Then, for the last computing unit (pp-last-stage), the scheduling process of the modified interleaved pipeline is shown in Figure 11:

[0125] S1101: First, determine whether the current step number step_id of the last computation unit is less than the total number of steps in the first chunk, first_chunk_steps. If so, proceed to S1102 for further determination. If step_id is not less than first_chunk_steps, proceed to step S1105. If step_id is not less than first_chunk_steps, proceed to S1102. If the determination result is yes, proceed to S1005.

[0126] S1102: Determine whether the last computation unit (pp-last-stage) is currently executing the last part of the interleaved schedule (or whether it is pp-stage 0 of the interleaved schedule). If not, place the computation result corresponding to the current step, such as a tensor, into fwd_buffer_queue. Continue with S1103. Otherwise, execute S1105.

[0127] S1103: Then check whether step_id is between left_id and right_id (if step_id is numbered from 0, the sending interval may include left_id but not right_id; if step_id is numbered from 1, the sending interval may include right_id but not left_id). If yes, execute S1104; if not, execute S1105.

[0128] S1104: Set the tensor of fwd_buffer_queue to a value that is not None. Then execute S1105.

[0129] S1105: Set tensor to None. Then execute S1106.

[0130] S1106. Determine whether tensor is None. If the tensor is not None, send the tensor in the bwd_buffer_queue to the first computing unit pp-stage0. If the tensor is None, do not send it.

[0131] For the scenario where the interleaved dimension num_model_chunks>2, please refer to Figure 8. The sending interval can include the forward calculation part or the backward calculation part between the curly brackets. For example, the last computing unit sends the tensors of micro-batch 1 of the first forward calculation to the first computing unit in step 2, sends the tensors of micro-batch 2 of the first forward calculation to the first computing unit in step 3..., and sends the tensors of micro-batch 5 of the second forward calculation to the first computing unit in step 11. For another example, the first computing unit sends the tensors of micro-batch 1 of the first backward calculation to the last computing unit in step 2, sends the tensors of micro-batch 2 of the first backward calculation to the last computing unit in step 3..., and sends the tensors of micro-batch 5 of the second backward calculation to the last computing unit in step 11.

[0132] FIG12 is a schematic structural diagram of a staggered pipeline scheduling device according to an embodiment of the present disclosure, comprising:

[0133] Parameter acquisition module 1201, used to obtain micro-batch scheduling parameters based on the pipeline segmentation dimension, interleaving dimension and accumulation number;

[0134] An interval determination module 1202 is configured to determine a sending interval of a micro-batch that requires cache scheduling based on the micro-batch scheduling parameters;

[0135] The cache scheduling module 1203 is configured to perform cache scheduling on the calculation results of the micro-batch that needs to be delayed in sending according to the sending interval and the cache unit.

[0136] In one embodiment, the pipeline splitting dimension includes the number of computing units used to perform pipeline scheduling on the model training process.

[0137] In one embodiment, the model training process is divided into multiple stages, and the interleaving dimension includes the number of stages that a computing unit needs to interleave for a micro-batch.

[0138] FIG13 is a schematic diagram of the structure of an interleaved pipeline scheduling device according to another embodiment of the present disclosure, which may include one or more features of the above-mentioned interleaved pipeline scheduling device. In one embodiment, the parameter acquisition module 1201 includes:

[0139] A first calculation submodule 1301 is configured to obtain the number of micro-batches included in the first block according to the pipeline segmentation dimension and the cumulative number of times;

[0140] A second calculation submodule 1302 is configured to obtain a total number of steps of the first block according to the number of micro-batches included in the first block and the interleaving dimension;

[0141] The third calculation submodule 1303 is configured to obtain the number of jump steps according to the pipeline segmentation dimension and the cumulative number of times.

[0142] In one embodiment, as shown in FIG13 , the interval determination module 1202 includes:

[0143] A first boundary determination submodule 1304 is configured to obtain a first boundary of the sending interval according to the jump step number, where the first boundary is used to determine a starting step number for sending the buffered data;

[0144] The second boundary determination submodule 1305 is configured to obtain a second boundary of the sending interval according to the first boundary, the number of micro-batches included in the first block, and the interleaving dimension. The second boundary is used to determine the number of steps at which the sending of the buffered data needs to be stopped.

[0145] In one embodiment, the cache scheduling module 1203 is further configured to perform cache scheduling on the calculation results of the micro-batch that needs to be delayed in the forward calculation process of the last computing unit according to the current step number of the micro-batch being processed by the last computing unit, the sending interval, and the forward cache queue.

[0146] In one embodiment, as shown in FIG13 , the cache scheduling module 1203 includes:

[0147] A first storage submodule 1306 is configured to store a tensor corresponding to the current step number in the forward cache queue if the current step number of the micro-batch being processed by the last computing unit is less than the total step number of the first block and the current step number is not the last interleaved step number;

[0148] The first sending submodule 1307 is configured to determine whether to send the head tensor in the forward buffer queue to the first computing unit according to the current step number and the sending interval.

[0149] In one embodiment, as shown in FIG13 , the cache scheduling module 1203 further includes:

[0150] A first setting submodule 1308 is configured to set the scheduling value of the head tensor in the forward buffer queue to a first value if the current step number is within the sending interval; or to set the scheduling value of the head tensor in the forward buffer queue to a second value if the current step number is not within the sending interval;

[0151] The first sending submodule 1307 is further configured to send the header tensor with the scheduling value being the first value to the first computing unit.

[0152] In one embodiment, as shown in FIG13 , the cache scheduling module 1203 further includes:

[0153] The second setting submodule 1309 is configured to set the scheduling value of the tensor corresponding to the current step number to a second value when the current step number is not less than the total step number of the first block.

[0154] In one embodiment, the cache scheduling module 1203 is further configured to perform cache scheduling on the calculation results of the micro-batch that needs to be delayed in the backward calculation process of the first computing unit according to the current number of steps of the micro-batch being processed by the first computing unit, the sending interval, and the backward cache queue.

[0155] In one embodiment, the cache scheduling module 1203 further includes:

[0156] A second storage submodule 1310 is configured to store a tensor corresponding to a current step number in the backward cache queue when the current step number of the micro-batch being processed by the first computing unit is less than the total step number of the first block and the current step number is not the last part of the interleaved step number;

[0157] The second sending submodule 1311 is configured to determine whether to send the head tensor in the backward buffer queue to the last computing unit according to the current step number and the sending interval.

[0158] In one embodiment, as shown in FIG13 , the cache scheduling module 1203 further includes:

[0159] A third setting submodule 1312 is configured to set the scheduling value of the head tensor in the backward buffer queue to the first value if the current step number is within the sending interval; or to set the scheduling value of the head tensor in the backward buffer queue to the second value if the current step number is not within the sending interval;

[0160] The second sending submodule 1311 is further configured to send the header tensor with the scheduling value being the first value to the last computing unit.

[0161] In one embodiment, as shown in FIG13 , the cache scheduling module 1203 further includes:

[0162] The fourth setting submodule 1313 is configured to set the scheduling value of the tensor corresponding to the current step number to a second value when the current step number is not less than the total step number of the first block.

[0163] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0164] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0165] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0166] FIG14 shows a schematic block diagram of an example electronic device 1400 that can be used to implement an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0167] As shown in FIG14 , device 1400 includes a computing unit 1401 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1402 or a computer program loaded from a storage unit 1408 into a random access memory (RAM) 1403. Various programs and data required for the operation of device 1400 may also be stored in RAM 1403. Computing unit 1401, ROM 1402, and RAM 1403 are connected to each other via a bus 1404. An input / output (I / O) interface 1405 is also connected to bus 1404.

[0168] Various components in device 1400 are connected to I / O interface 1405, including: an input unit 1406, such as a keyboard, mouse, etc.; an output unit 1407, such as various types of displays, speakers, etc.; a storage unit 1408, such as a magnetic disk, optical disk, etc.; and a communication unit 1409, such as a network card, modem, wireless communication transceiver, etc. Communication unit 1409 allows device 1400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0169] The computing unit 1401 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1401 performs the various methods and processes described above, such as the interleaved pipeline scheduling method. For example, in some embodiments, the interleaved pipeline scheduling method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 1408. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1400 via the ROM 1402 and / or the communication unit 1409. When the computer program is loaded into the RAM 1403 and executed by the computing unit 1401, one or more steps of the interleaved pipeline scheduling method described above can be performed. Alternatively, in other embodiments, the computing unit 1401 may be configured to execute the staggered pipeline scheduling method in any other appropriate manner (eg, by means of firmware).

[0170] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0171] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0172] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0173] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0174] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0175] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0176] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0177] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A staggered pipeline scheduling method, comprising: According to the pipeline segmentation dimension, interleaving dimension and accumulation times, the micro-batch scheduling parameters are obtained; Determining a sending interval of a micro-batch that needs to be cached and scheduled according to the micro-batch scheduling parameters; According to the sending interval and the cache unit, cache scheduling is performed on the calculation results of the micro-batch that needs to be delayed in sending.

2. The method according to claim 1, wherein The pipeline splitting dimension includes the number of computing units used to perform pipeline scheduling on the model training process.

3. The method according to claim 1 or 2, wherein: The model training process is divided into multiple stages, and the interleaving dimension includes the number of stages that a computing unit needs to interleave for a micro-batch.

4. The method according to any one of claims 1 to 3, wherein According to the pipeline segmentation dimension, interleaving dimension and accumulation times, the micro-batch scheduling parameters are obtained, including: Obtaining the number of micro-batches included in the first block according to the pipeline segmentation dimension and the number of accumulations; Obtaining a total number of steps for the first block according to the number of micro-batches included in the first block and the interleaving dimension; The number of jump steps is obtained according to the pipeline segmentation dimension and the cumulative number of times.

5. The method according to claim 4, wherein Determining a sending interval of a micro-batch requiring cache scheduling according to the micro-batch scheduling parameters includes: Obtaining a first boundary of the sending interval according to the jump step number, where the first boundary is used to determine a starting step number for sending buffered data; A second boundary of the sending interval is obtained according to the first boundary, the number of micro-batches included in the first block, and the interleaving dimension, and the second boundary is used to determine the number of steps at which sending of the cached data needs to be stopped.

6. The method according to claim 4 or 5, wherein: Performing cache scheduling on the calculation results of the micro-batch that needs to be delayed according to the sending interval and the cache unit, including: According to the current step number of the micro-batch being processed by the last computing unit, the sending interval and the forward cache queue, cache scheduling is performed on the calculation results of the micro-batch that needs to be delayed in the forward calculation process of the last computing unit.

7. The method according to claim 6, wherein: According to the current step number of the micro-batch being processed by the last computing unit, the sending interval, and the forward cache queue, cache scheduling is performed on the calculation results of the micro-batch that needs to be delayed in the forward calculation process of the last computing unit, including: When a current step number of the micro-batch being processed by the last computation unit is less than a total step number of the first block, and the current step number is not a last part of the interleaved step number, storing a tensor corresponding to the current step number in the forward cache queue; Determine whether to send a head tensor in the forward buffer queue to a first computing unit according to the current step number and the sending interval.

8. The method according to claim 7, wherein: Determining, according to the current step number and the sending interval, whether to send a head tensor in the forward buffer queue to a first computing unit includes: If the current step number is within the sending interval, the scheduling value of the head tensor in the forward cache queue is set to a first value; or if the current step number is not within the sending interval, the scheduling value of the head tensor in the forward cache queue is set to a second value; The header tensor whose scheduling value is a first value is sent to the first computation unit.

9. The method according to any one of claims 6 to 8, wherein According to the current step number of the micro-batch being processed by the last computing unit, the sending interval, and the forward cache queue, cache scheduling is performed on the calculation results of the micro-batch that needs to be delayed in the forward calculation process of the last computing unit, including: When the current number of steps is not less than the total number of steps of the first block, the scheduling value of the tensor corresponding to the current number of steps is set to a second value.

10. The method according to claim 4 or 5, wherein: Performing cache scheduling on the calculation results of the micro-batch that needs to be delayed according to the sending interval and the cache unit, including: According to the current step number of the micro-batch being processed by the first computing unit, the sending interval and the backward cache queue, cache scheduling is performed on the calculation results of the micro-batch that needs to be delayed in the backward calculation process of the first computing unit.

11. The method according to claim 10, wherein: According to the current step number of the micro-batch being processed by the first computing unit, the sending interval, and the backward cache queue, cache scheduling is performed on the calculation results of the micro-batch that needs to be delayed in the backward calculation process of the first computing unit, including: When a current step number of the micro-batch being processed by the first computation unit is less than a total step number of the first block, and the current step number is not a last part of the interleaved step number, storing a tensor corresponding to the current step number in the backward buffer queue; Determine whether to send the head tensor in the backward buffer queue to the last computing unit according to the current step number and the sending interval.

12. The method according to claim 11, wherein Determining whether to send the head tensor in the backward buffer queue to the last computing unit according to the current step number and the sending interval includes: When the current step number is within the sending interval, the scheduling value of the head tensor in the backward buffer queue is set to a first value; or, when the current step number is not within the sending interval, the scheduling value of the head tensor in the backward buffer queue is set to a second value; The head tensor whose scheduling value is a first value is sent to the last computing unit.

13. The method according to any one of claims 10 to 12, wherein According to the current step number of the micro-batch being processed by the first computing unit, the sending interval, and the forward cache queue, cache scheduling is performed for the calculation results of the micro-batch that needs to be delayed in the backward calculation process of the last computing unit, including: When the current number of steps is not less than the total number of steps of the first block, the scheduling value of the tensor corresponding to the current number of steps is set to a second value.

14. A staggered pipeline scheduling device comprising: The parameter acquisition module is used to obtain the micro-batch scheduling parameters based on the pipeline segmentation dimension, interleaving dimension, and accumulation number; An interval determination module, configured to determine a sending interval of a micro-batch that requires cache scheduling according to the micro-batch scheduling parameters; The cache scheduling module is used to perform cache scheduling on the calculation results of the micro-batch that needs to be delayed in sending according to the sending interval and the cache unit.

15. The device according to claim 14, wherein The pipeline splitting dimension includes the number of computing units used to perform pipeline scheduling on the model training process.

16. The device according to claim 14 or 15, wherein The model training process is divided into multiple stages, and the interleaving dimension includes the number of stages that a computing unit needs to interleave for a micro-batch.

17. The device according to any one of claims 14 to 16, wherein The parameter acquisition module includes: A first calculation submodule is configured to obtain the number of micro-batches included in the first block according to the pipeline segmentation dimension and the number of accumulations; A second calculation submodule, configured to obtain a total number of steps of the first block according to the number of micro-batches included in the first block and the interleaving dimension; The third calculation submodule is used to obtain the number of jump steps according to the pipeline segmentation dimension and the cumulative number of times.

18. The device according to claim 17, wherein The interval determination module includes: a first boundary determination submodule, configured to obtain a first boundary of the sending interval according to the jump step number, wherein the first boundary is used to determine a starting step number for sending buffered data; A second boundary determination submodule is configured to obtain a second boundary of the sending interval according to the first boundary, the cumulative number of micro-batches included in the first block, and the interleaving dimension, wherein the second boundary is used to determine the number of steps at which the sending of the cached data needs to be stopped.

19. The device according to claim 17 or 18, wherein The cache scheduling module is further configured to perform cache scheduling on the calculation results of the micro-batch that needs to be delayed in the forward calculation process of the last computing unit according to the current step number of the micro-batch being processed by the last computing unit, the sending interval, and the forward cache queue.

20. The device according to claim 19, wherein The cache scheduling module includes: A first storage submodule is configured to, when the current step number of the micro-batch being processed by the last computing unit is less than the total step number of the first block, and the current step number is not the last part of the interleaved step number, Storing the tensor corresponding to the current step number into the forward cache queue; The first sending submodule is configured to determine, based on the current step number and the sending interval, whether to send the head tensor in the forward cache queue to the first computing unit.

21. The device according to claim 20, wherein The cache scheduling module also includes: a first setting submodule, configured to set the scheduling value of the head tensor in the forward cache queue to a first value when the current step number is within the sending interval; or to set the scheduling value of the head tensor in the forward cache queue to a second value when the current step number is not within the sending interval; The first sending submodule is further configured to send the header tensor with the scheduling value being the first value to the first computing unit.

22. The device according to any one of claims 19 to 21, wherein The cache scheduling module further includes: The second setting submodule is used to set the scheduling value of the tensor corresponding to the current step number to a second value when the current step number is not less than the total step number of the first block.

23. The device according to claim 17 or 18, wherein The cache scheduling module is further configured to perform cache scheduling on calculation results of the micro-batch that needs to be delayed in sending during the backward calculation process of the first computing unit according to the current number of steps of the micro-batch being processed by the first computing unit, the sending interval, and the backward cache queue.

24. The device according to claim 23, wherein The cache scheduling module further includes: a second storage submodule, configured to store a tensor corresponding to the current step number in the backward cache queue when the current step number of the micro-batch being processed by the first computing unit is less than the total step number of the first block and the current step number is not the last part of the interleaved step number; The second sending submodule is used to determine whether to send the head tensor in the backward buffer queue to the last computing unit according to the current step number and the sending interval.

25. The apparatus according to claim 24, wherein The cache scheduling module also includes: a third setting submodule, configured to set the scheduling value of the head tensor in the backward cache queue to a first value when the current step number is within the sending interval; or to set the scheduling value of the head tensor in the backward cache queue to a second value when the current step number is not within the sending interval; The second sending submodule is further configured to send the header tensor whose scheduling value is the first value to the last computing unit.

26. The device according to any one of claims 23 to 25, wherein The cache scheduling module further includes: The fourth setting submodule is used to set the scheduling value of the tensor corresponding to the current step number to a second value when the current step number is not less than the total step number of the first block.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 13.

29. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images