Interleaved pipeline scheduling method, apparatus, electronic device, storage medium, and program

Abstract

This disclosure provides interleaved pipeline scheduling methods, apparatus, devices, and storage media, particularly relating to the computer technology field, and more particularly to the artificial intelligence technology field, including deep learning, neural networks, and parallel computing. [Solution] A specific solution includes obtaining micro-batch scheduling parameters based on the pipeline splitting dimension, interleave dimension, and cumulative count; determining the transmission interval of a micro-batch requiring cache scheduling based on the micro-batch scheduling parameters; and performing cache scheduling on the computation results of the micro-batch requiring delayed transmission based on the transmission interval and cache units. According to this disclosure, a cache unit, for example, a cache queue, can be created, the computation results of some computation units can be cached in the cache unit, and the computation results can be transmitted with a delay. This allows for delayed transmission of the computation results of some micro-batches, improving the applicability of interleaved pipeline scheduling.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular, to the field of artificial intelligence technologies such as deep learning, neural networks, and parallel computing.

Background Art

[0002] In the field of deep learning, a distributed strategy can be used to improve the computing efficiency of computing units. The pipeline parallel strategy is one of the commonly used distributed strategies. Here, the interleaved pipeline can further optimize the bubble occupancy rate of the normal pipeline.

Summary of the Invention

Problems to be Solved by the Invention

[0003] This disclosure provides an interleaved pipeline scheduling method, apparatus, device, and storage medium.

Means for Solving the Problems

[0004] In one aspect of this disclosure, an interleaved pipeline scheduling method is provided, and the method includes: acquiring micro-batch scheduling parameters based on a pipeline splitting dimension, an interleaving dimension, and a cumulative number of times; determining a transmission interval of a micro-batch that requires cache scheduling based on the micro-batch scheduling parameters; performing cache scheduling on a calculation result of a micro-batch that requires delayed transmission based on the transmission interval and a cache unit.

[0005] In another aspect of this disclosure, an interleaved pipeline scheduling apparatus is provided, and the apparatus includes: A parameter acquisition module for obtaining micro-batch scheduling parameters based on pipeline splitting dimension, interleave dimension, and cumulative count, A spatial determination module for determining the transmission interval of a microbatch that requires cache scheduling based on the microbatch scheduling parameters, The system includes a cache scheduling module for performing cache scheduling on the calculation results of microbatches requiring delayed transmission, based on the transmission interval and cache units.

[0006] In another aspect of this disclosure, an electronic device is provided, which is At least one processor, The system comprises at least one processor and memory that is communicated with, The memory stores instructions that are executable by the at least one processor, and when executed by the at least one processor, the instructions cause one of the methods in the embodiments of the present disclosure to be performed.

[0007] Another aspect of the present disclosure provides a non-temporary computer-readable storage medium that stores computer instructions for causing a computer to perform any one of the methods of the embodiments of the present disclosure.

[0008] In another aspect of the present disclosure, a program product is provided which, when executed by a processor, includes a program for performing any of the methods in the embodiments of the present disclosure.

[0009] In embodiments of this disclosure, a cache mechanism is introduced to assist in the scheduling of interleaved pipelines, and the transmission interval of microbatches requiring cache scheduling is determined through microbatch scheduling parameters. This allows for delayed transmission of the calculation results of some microbatches, thereby improving the applicability of interleaved pipeline scheduling without limiting the number of microbatches scheduled at once.

[0010] It should be understood that the content contained herein is not intended to describe any key points or important features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Further details of other features of this disclosure are provided in the specification below.

[0011] The attached drawings are for the purpose of better understanding the solutions of this disclosure and do not constitute a limitation of this disclosure. [Brief explanation of the drawing]

[0012] [Figure 1] This is a schematic flowchart of an interleaved pipeline scheduling method according to one embodiment of the present disclosure. [Figure 2] This is a schematic diagram of the pipeline scheduling scenario described in this disclosure. [Figure 3] This is a schematic diagram of the interleaved pipeline scheduling scenario described in this disclosure. [Figure 4] This is a schematic flowchart of an interleaved pipeline scheduling method according to another embodiment of the present disclosure. [Figure 5] This is a schematic flowchart of an interleaved pipeline scheduling method according to another embodiment of the present disclosure. [Figure 6] This is a schematic flowchart of an interleaved pipeline scheduling method according to another embodiment of the present disclosure. [Figure 7] This is a schematic diagram of the interleaved pipeline scheduling scenario described herein. [Figure 8] This is a schematic diagram of the interleaved pipeline scheduling scenario described herein. [Figure 9] This is a schematic flowchart of an interleaved pipeline scheduling method according to another embodiment of the present disclosure. [Figure 10] This is a schematic diagram of the interleaved pipeline scheduling for reverse calculations. [Figure 11]It is a flow schematic diagram of interleaved pipeline scheduling for forward calculation. [Figure 12] It is a schematic diagram showing the configuration of an interleaved pipeline scheduling apparatus according to an embodiment of the present disclosure. [Figure 13] It is a schematic diagram showing the configuration of an interleaved pipeline scheduling apparatus according to another embodiment of the present disclosure. [Figure 14] It is a block diagram of an electronic device for realizing an embodiment of the present disclosure.

Mode for Carrying Out the Invention

[0013] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the accompanying drawings. These drawings include various details of the embodiments of the present disclosure for the purpose of assisting understanding, and these should be considered to be merely exemplary. Therefore, those skilled in the art should understand that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, descriptions of well-known features and structures are omitted in the following description for the sake of clarity and conciseness.

[0014] FIG. 1 is a flow schematic diagram of an interleaved pipeline scheduling method according to an embodiment of the present disclosure, and this method includes: In S101, obtaining micro-batch scheduling parameters based on a pipeline division dimension, an interleaved dimension, and a cumulative number of times; In S102, determining a transmission interval of a micro-batch that requires cache scheduling based on the micro-batch scheduling parameters; In S103, performing cache scheduling on a calculation result of a micro-batch that requires delayed transmission based on the transmission interval and a cache unit.

[0015] In the embodiments of this disclosure, artificial intelligence models such as neural networks require significant computing power during the training process, and therefore can be processed in parallel using multiple computing units. Different stages of the model can be computed through different computing units, thereby achieving pipeline parallel computing. In pipeline parallel computing, data can be transferred between adjacent computing units via a communication link. An example of a pipeline scheduling scenario is shown in Figure 2. Here, the horizontal axis represents the number of computation steps for one computing unit, and in this pipeline scheduling scenario, one computing unit can compute one forward stage and one reverse stage of the model. Rectangles correspond to the number of steps in the forward computation, and squares correspond to the number of steps in the reverse computation. Assuming that eight microbatches need to be computed, this can be divided into two blocks. First, four computing units are used to sequentially compute the microbatches of the first block (numbers 1-4), and then the microbatches of the second block (numbers 5-8) are sequentially computed.

[0016] In the embodiments of this disclosure, each computing unit in interleaved pipeline scheduling can perform computing functions for multiple stages of the model training process. An example of an interleaved pipeline scheduling scenario is shown in Figure 3. The number of interleaves in this interleaved pipeline scheduling scenario is 2. One computing unit can compute multiple forward stages and multiple reverse stages of the model. After the first to fourth computing units sequentially perform forward calculations on a microbatch, the fourth computing unit must return the calculation results to the first computing unit and perform forward calculations again. For example, the first computing unit performs the first forward calculation on microbatch 1 in step 1 and the second forward calculation on microbatch 1 in step 5. Similarly, in the reverse direction, the first computing unit must return the reverse calculation results to the fourth computing unit and perform reverse calculations again, and so on.

[0017] Because each computing unit directly transmits its results to subsequent computing units after computation, interleaved pipeline parallel strategies typically require limiting the batch size, for example, to an integer multiple of the pipeline partition dimension. Many batch sizes cannot meet this requirement, limiting the use cases for interleaved pipeline scheduling.

[0018] In embodiments of this disclosure, a cache unit, such as a cache queue, can be created to cache the computation results of some compute units and transmit the computation results with a delay. This removes the limitations on batch size and allows for flexible support of different batch sizes in pipeline scheduling. For example, the forward computation result of the last compute unit can be cached in the cache unit and transferred to the first compute unit after a certain number of steps. Alternatively, the reverse computation result of the first compute unit can be cached in the cache unit and transferred to the last compute unit after a certain number of steps.

[0019] In embodiments of this disclosure, microbatch scheduling parameters can be calculated and obtained based on the pipeline partitioning dimension, the interleaving dimension, and the cumulative count for delayed transmission of computation results in the cache unit. Here, the pipeline partitioning dimension can be determined based on the number of compute units used. For example, if there are four compute units, the pipeline partitioning dimension is also four. The sample set of the model to be trained includes multiple microbatches, each microbatch may contain one or more samples. The cumulative count can be determined based on the number of microbatches that can be processed in a single interleaved pipeline scheduling. For example, if there are nine microbatches that can be processed in a single interleaved pipeline scheduling, the cumulative count is also nine. If there are ten microbatches that can be processed in a single interleaved pipeline scheduling, the cumulative count is also ten. If the number of microbatches that can be processed in a single interleaved pipeline scheduling does not satisfy an integer multiple of the pipeline partitioning dimension, some of the data that needs to be transferred over the communication link can be cached. The interleaving dimension may include the number of stages in the model training process for which one compute unit requires interleaved execution for one microbatch. For example, one computing unit performs one stage of forward computation for one microbatch in step 1, and another stage of forward computation for the same microbatch in step 6. In this case, the interleaved dimension can be 2. Microbatch scheduling parameters can be used to determine the transmission interval of microbatches that require cache scheduling in a single interleaved pipeline scheduling. This transmission interval can be used for delayed transmission of data (the computation results of the microbatch) stored in the cache unit.

[0020] According to the embodiments of this disclosure, a cache mechanism is introduced to assist in the scheduling of interleaved pipelines, and the transmission interval of microbatches requiring cache scheduling is determined through microbatch scheduling parameters. This allows for delayed transmission of the calculation results of some microbatches, without limiting the number of microbatches scheduled at once, thereby improving the applicability of interleaved pipeline scheduling.

[0021] In one embodiment, the pipeline partitioning dimension includes the number of computing units used to perform pipeline scheduling for the model training process.

[0022] In embodiments of this disclosure, the size of the pipeline partitioning dimension can be determined based on the number of compute units performing pipeline scheduling for the model training process. For example, one compute unit may include one or more graphics processing units (GPUs), and one GPU may be divided into multiple compute units (or one compute unit may include some of the compute capabilities of one GPU). The size of the pipeline partitioning dimension can be determined based on the number of available compute units.

[0023] According to the embodiments of this disclosure, the size of the pipeline partitioning dimension can be determined in combination with the actual number of computing units, which is advantageous for rationally utilizing computing units and improving the computing power of interleaved pipeline scheduling.

[0024] In one embodiment, the model training process is divided into multiple stages, and the interleaving dimension includes the number of stages for which one computing unit requires interleaving execution for one microbatch.

[0025] In embodiments of this disclosure, the model training process can be divided into multiple stages, and the specific method of division can be determined by referring to the structure and functionality of the model. The size of the interleaved dimension is related to the number of computing units and the number of stages into which the model training process is divided. For example, if the model training process is divided into eight stages and the number of computing units is four, the size of the interleaved dimension can be two. In this case, one computing unit requires interleaved execution for two stages of computation of the model for one microbatch. The first computing unit performs the computation of stages 1 and 5, the second computing unit performs the computation of stages 2 and 6, the first computing unit performs the computation of stages 3 and 7, and the first computing unit performs the computation of stages 4 and 8.

[0026] According to the embodiments of this disclosure, the size of the pipeline interleaved dimension can be determined in combination with the number of partitioned stages in the model training process, which is advantageous for improving the computational efficiency of interleaved pipeline scheduling.

[0027] Figure 4 is a schematic flow diagram of an interleaved pipeline scheduling method according to another embodiment of the present disclosure, the method may include one or more features of the above interleaved pipeline scheduling method. In one embodiment, micro-batch scheduling parameters are obtained based on the pipeline splitting dimension, interleaved dimension and cumulative count of S101, In S401, the number of microbatches included in the first subblock is obtained based on the pipeline partitioning dimension and the cumulative count, In S402, the total number of steps in the first subblock is obtained based on the number of microbatches included in the first subblock and the interleaved dimension. S403 includes obtaining the number of jump steps based on the pipeline division dimension and the cumulative number.

[0028] In the embodiments of this disclosure, the total number of microbatches included in a single interleaved pipeline scheduling does not have to be an integer multiple of the pipeline partitioning dimension. In this case, subblocks are performed on the microbatches included in a single interleaved pipeline scheduling. For example, the number of microbatches included in a subblock that is not an integer multiple of the pipeline partitioning dimension (which can be called the first subblock) can be calculated and obtained based on the pipeline partitioning dimension and the cumulative count. For example, the number of microbatches included in the first subblock can be obtained by dividing the cumulative count by the partitioning dimension, finding the remainder, and adding the remainder to the partitioning dimension. Of the microbatches included in a single interleaved pipeline scheduling, the remaining microbatches, excluding those included in the first subblock, can have their calculation results sent directly without using delayed transmission of calculation results with a cache. For example, in a single interleaved pipeline scheduling, if the partitioning dimension is 4 and the cumulative count is 9, the number of microbatches included in the first subblock is (9%4)+4=5. Therefore, microbatches 1, 2, 3, 4, and 5 of the first subblock require cache scheduling, while the remaining microbatches 6, 7, 8, and 9 undergo normal scheduling.

[0029] In embodiments of the present disclosure, the total number of steps that a first subblock needs to perform in one compute unit can be obtained based on the number of microbatches contained in the first subblock and the interleaved dimension. This total number of steps may include the total number of steps in forward computation and / or the total number of steps in reverse computation. For example, the product of the number of microbatches contained in the first subblock and the interleaved dimension can be used as the total number of steps for the first subblock. If the number of microbatches contained in the first subblock is 5 and the interleaved dimension is 2, the total number of steps for the first subblock is 5 * 2 = 10, and each compute unit will perform a total of 10 forward computations and a total of 10 reverse computations for all microbatches in the first subblock.

[0030] In embodiments of this disclosure, the jump step number can be used to determine the number of steps required between storing the computation result of one microbatch of the first subblock in the cache queue and sending the result with a delay. The remainder obtained by dividing the cumulative count by the pipeline partitioning dimension can be used as the jump step number. For example, if the partitioning dimension is 4 and the cumulative count is 9, the jump step number is 9%4=1, in which case one step is required between sending the computation result of one microbatch of the first subblock from the cache.

[0031] According to the embodiments of this disclosure, by calculating microbatch scheduling parameters such as the number of microbatches included in the first subblock, the total number of steps in the first subblock, and the number of jump steps, the calculation results of the microbatches included in the first subblock in the cache queue can be transmitted with a delay.

[0032] Figure 5 is a schematic flow diagram of an interleaved pipeline scheduling method according to another embodiment of the present disclosure, the method may include one or more features of the above interleaved pipeline scheduling method. In one embodiment, based on the microbatch scheduling parameters in S102, determining the transmission interval of a microbatch that requires cache scheduling is: In S501, the first boundary of the transmission section is obtained based on the number of jump steps, and this first boundary is used to determine the starting number of steps for which cached data should be transmitted. In S502, a second boundary of the transmission interval is obtained based on the first boundary, the number of microbatches included in the first subblock, and the interleave dimension, and the second boundary is used to determine the number of steps for which transmission of cached data should be stopped.

[0033] In embodiments of this disclosure, a transmission interval can be understood as a number of steps in which data in a cache unit needs to be transmitted to other computing units. A transmission interval may have a first boundary and a second boundary. The first boundary, also called the left boundary, can be used to determine the number of steps in which the transmission of microbatch computation results in the cache queue needs to begin. The second boundary, also called the right boundary, can be used to determine the number of steps in which the transmission of microbatch computation results in the cache queue needs to begin. A transmission interval can be a closed interval, a half-open interval, or an open interval. For example, if the number of jump steps is used as the first boundary of the transmission interval, the transmission interval does not include the first boundary itself. Also, if the number of jump steps plus 1 is used as the first boundary of the transmission interval, the transmission interval includes the first boundary itself.

[0034] In embodiments of this disclosure, a second boundary of a transmission interval can be calculated based on the first boundary, the number of microbatches contained in the first subblock, and the interleaved dimension. For example, the second boundary is obtained by multiplying the difference obtained by subtracting 1 from the interleaved dimension by the number of microbatches contained in the first subblock, and adding the first boundary to the resulting product. For example, if the first boundary is 1, the number of microbatches contained in the first subblock is 5, and the interleaved dimension is 2, the second boundary is calculated as 1 + 5 * (2 - 1) = 6.

[0035] In embodiments of this disclosure, the relationship between the transmit interval and the cache interval can be determined based on the number of jump steps. The cache interval may include the number of steps required to store the results of a microbatch in the cache queue. The difference between the boundary of the transmit interval and the cache interval can be equal to or calculated based on the number of jump steps. For example, the cache interval is [1,6) and the transmit interval is [2,7). Or, the cache interval is [1,6) and the transmit interval is (1,6).

[0036] According to embodiments of this disclosure, the transmission of data in a cache queue can be controlled through the boundaries of the transmission interval, thereby enabling delayed transmission of the computation results of each microbatch in the first subblock to the corresponding computing unit.

[0037] Figure 6 is a schematic flow diagram of an interleaved pipeline scheduling method according to another embodiment of the present disclosure, which may include one or more features of the above interleaved pipeline scheduling method. In one embodiment, the cache unit may include a cache queue, for example, a forward cache queue. Based on the transmission interval and the cache unit, cache scheduling can be performed for the computation results of microbatches requiring delayed transmission. The last computing unit performs cache scheduling for the computation results of microbatches that require delayed transmission in the forward computation process, based on the current number of steps of the microbatch being processed by the last computing unit, the transmission interval, and the forward cache queue.

[0038] In embodiments of this disclosure, a forward cache queue (or forward compute cache queue, forward propagation cache queue, etc.) can be used to store the computation results of microbatches in a forward compute process, such as tensors. In a forward compute process, it is necessary to first cache the computation results of the last compute unit and then transmit them to the first compute unit with a delay after a number of jump steps. In this process, the computation results of the microbatch of the last compute unit can first be stored in the forward cache queue based on the current number of steps, and then, after comparing the current number of steps with the transmission interval and setting an interval of the number of jump steps, the computation results stored in the forward cache queue can be transmitted to the first compute unit.

[0039] According to the embodiments of this disclosure, interleaved pipeline scheduling can be performed on the training sample set in the model training process through cache queues and transmission intervals, eliminating the need to restrict the relationship between the number of microbatches of the training sample set and the partitioning dimension, thereby improving the applicability of interleaved pipeline scheduling.

[0040] In one embodiment, the last computing unit performs cache scheduling for the computation results of microbatches that require delayed transmission in the forward computing process, based on the current number of steps of the microbatch being processed, the transmission interval, and the forward cache queue. In S601, if the current number of steps in the microbatch being processed by the last computing unit is less than the total number of steps in the first subblock, and the current number of steps is not the number of interleaved steps in the last part, then the tensor corresponding to the current number of steps is stored in the forward cache queue. S602 includes determining whether to send the head tensor in the forward cache queue to the first compute unit based on the current step number and the transmission interval.

[0041] In the embodiments of this disclosure, for forward computation, if the number of computation steps currently being executed by the last computation unit (i.e., the current number of steps) is less than the total number of steps in the first subblock, the current number of steps represents the number of steps being performed on microbatches within the first subblock. If the current number of steps is equal to or greater than the total number of steps in the first subblock, the current number of steps represents the number of steps being performed on other microbatches prior to the first subblock, in which case it is not necessary to cache the computation results of the microbatches.

[0042] If the current step count is less than the total number of steps in the first subblock, it can be further determined whether the current step count is the interleaved step count of the last part. The interleaved step count of the last part is related to the interleaved dimension VPP. For example, the first threshold is the product of the difference obtained by subtracting 1 from VPP and multiplying it by the number of microbatches contained in the first subblock, i.e., (VPP-1)*number of microbatches in the first subblock. If the current step count is greater than the first threshold, it indicates that the current step count is the interleaved step count of the last part. If the current step count is not the interleaved step count of the last part, the tensor of the step count is stored in the forward cache queue. Referring to Figure 7, the current step count of the last computing unit can start accumulating from 1 (or 0), i.e., the current step count of the first step is 1, and the second step is 2. If the number of microbatches contained in the first subblock is 5 and the interleaved dimension is 2, the first threshold is 5. If the current number of steps in the last computing unit is between 1 and 5, it is not the number of interleaved steps in the last part; however, if the current number of steps is between 6 and 10, it is the number of interleaved steps in the last part. Referring to Figure 8, if the number of microbatches in the first subblock is 5 and the interleaved dimension is 3, the first threshold is 10. If the current number of steps in the last computing unit is between 1 and 11, it is not the number of interleaved steps in the last part; however, if the current number of steps is between 11 and 15, it is the number of interleaved steps in the last part.

[0043] In the embodiments of this disclosure, the period from the start step number of the first subblock to the step number corresponding to the first threshold can be understood as a cache interval. Unlike the transmission interval, the cache interval allows for delayed transmission of data in the cache queue. For example, the current step number of the last compute unit to compute microbatch 1 is 1, and since the current step number is less than the first threshold, the compute result of microbatch 1 can be stored in the first item (head) of the forward cache queue. However, since the current step number is not included in the transmission interval, no data in the forward cache queue is transmitted. The current step number of the last compute unit to compute microbatch 2 is 2, and since the current step number is less than the first threshold, the compute result of microbatch 2 can be stored in the second item of the forward cache queue. Since the current step number is included in the transmission interval, the head data in the forward cache queue can be sent to the first compute unit, changing the second item in the forward cache queue to the first item. Alternatively, the head data in the forward cache queue can be sent to the first compute unit first, and then the compute result of microbatch 2 can be cached.

[0044] According to an embodiment of the present disclosure, after the last computing unit has computed a microbatch tensor and stored it in a forward cache queue, it is possible to determine, based on the current step count and transmission interval, whether to send the head tensor in the forward cache queue to the first computing unit at the current step count, enabling delayed transmission of cached tensors and increasing the number of microbatches to which interleaved pipeline scheduling can be applied.

[0045] In one embodiment, determining whether to send the head tensor in the forward cache queue to the first compute unit based on the current step number and the transmission interval is: If the current number of steps falls within the transmission interval, the scheduling value of the head tensor in the forward cache queue is set to the first value; or, if the current number of steps falls outside the transmission interval, the scheduling value of the head tensor in the forward cache queue is set to the second value. This includes sending the head tensor, whose scheduling value is the first value, to the first computing unit.

[0046] In the embodiments of this disclosure, the first and second values ​​are different. For example, the first value can be a non-null value, and the second value can be a null value (none). If the current step count is within the transmission interval, the scheduling value of the head tensor in the forward cache can be set to a non-null value, for example, 1. If the current step count is not within the transmission interval, the scheduling value of the head tensor in the forward cache queue is set to a null value, for example, none. For example, if the number of microbatches in the first subblock is 5, the interleaved dimension is 2, the current step count starts accumulating from 1, and the transmission interval is (1,6), then first, the tensor of microbatch 1 calculated and obtained at the current step count of 1 is stored in the head of the forward cache queue. Next, since the current step count of 1 is not in the transmission interval (1,6), the scheduling value of the head tensor in the forward cache queue is set to null. Because the scheduling value of the head tensor is null, the current step count of 1 does not transmit the head tensor. Subsequently, the current step count accumulates to 2, and the current step count of 2 is in the transmission interval (1,6), so the scheduling value of the head tensor in the forward cache queue is set to 1. Because the scheduling value of the head tensor is 1, the head tensor (the tensor of microbatch 1) is transmitted at the current step count of 2. After transmission, the tensor of microbatch 1 can be deleted, and the tensor of microbatch 2 can be made the head tensor in the forward cache queue.

[0047] According to embodiments of this disclosure, the timing of transmission from the last compute unit to the first compute unit can be controlled through the scheduling value of the head tensor in the forward cache queue, thereby delaying the transmission of some microbatch tensors and increasing the number of microbatch to which interleaved pipeline scheduling can be applied.

[0048] In one embodiment, the last computing unit performs cache scheduling for the computation results of microbatches that require delayed transmission in the forward computing process, based on the current number of steps of the microbatch being processed, the transmission interval, and the forward cache queue. If the current number of steps is equal to or greater than the total number of steps in the first subblock, this includes setting the scheduling value of the tensor corresponding to the current number of steps to the second value.

[0049] In embodiments of this disclosure, if the current step count is greater than or equal to the total number of steps in the first subblock, the computation to be performed corresponds to other subblocks that do not require cache scheduling, and the scheduling value of the tensor corresponding to the current step count can be set to a second value, such as a null value, and there is no need to store the tensor corresponding to the current step count in a forward cache queue or to perform delayed transmission.

[0050] According to the embodiments of this disclosure, other subblocks that do not require cache scheduling can be normally transmitted, thereby reducing scheduling complexity and improving computational efficiency.

[0051] Figure 9 is a schematic flow diagram of an interleaved pipeline scheduling method according to another embodiment of the present disclosure, which may include one or more features of the above interleaved pipeline scheduling method. In one embodiment, the cache unit may include a reverse cache queue, and cache scheduling for the computation results of microbatches requiring delayed transmission based on the transmission interval and the cache unit includes the first compute unit performing cache scheduling for the computation results of microbatches requiring delayed transmission in the reverse compute process based on the current number of steps of the microbatch being processed by the first compute unit, the transmission interval and the reverse cache queue.

[0052] In embodiments of this disclosure, a reverse cache queue (or reverse compute cache queue, reverse propagation cache queue, etc.) can be used to store the computation results of microbatches in a reverse compute process, such as tensors. In a reverse compute process, it is necessary to first cache the computation results of the first compute unit and then transmit them to the last compute unit with a delay after a number of jump steps. In this process, the computation results of the microbatch of the first compute unit can first be stored in the reverse cache queue based on the current number of steps, and then, after comparing the current number of steps with the transmission interval and setting an interval of the number of jump steps, the computation results stored in the reverse cache queue can be transmitted to the last compute unit.

[0053] According to the embodiments of this disclosure, interleaved pipeline scheduling can be performed on the training sample set in the model training process through cache queues and transmission intervals, eliminating the need to restrict the relationship between the number of microbatches of the training sample set and the partitioning dimension, thereby improving the applicability of interleaved pipeline scheduling.

[0054] In one embodiment, the first computing unit performs cache scheduling for the computation results of microbatches that require delayed transmission in the reverse computation process, based on the current number of steps of the microbatch being processed, the transmission interval, and the reverse cache queue. In S901, if the current number of steps in the microbatch being processed by the first computing unit is less than the total number of steps in the first subblock, and the current number of steps is not the number of interleaved steps in the last part, then the tensor corresponding to the current number of steps is stored in the reverse cache queue. S902 includes determining whether to send the head tensor in the reverse cache queue to the last compute unit based on the current step number and the transmission interval.

[0055] In the embodiments of this disclosure, for reverse computation, if the number of computation steps currently being executed by the first computation unit (i.e., the current number of steps) is less than the total number of steps in the first subblock, the current number of steps represents the number of steps to perform computation on microbatches within the first subblock. If the current number of steps is equal to or greater than the total number of steps in the first subblock, the current number of steps represents the number of steps to perform computation on other microbatches prior to the first subblock, in which case it is not necessary to cache the computation results of the microbatches.

[0056] If the current step count is less than the total number of steps in the first subblock, it can be further determined whether the current step count is the interleaved step count for the last part. The interleaved step count for the last part is related to the interleaved dimension VPP. For example, the first threshold is the product of the difference obtained by subtracting 1 from VPP and multiplying it by the number of microbatches contained in the first subblock. If the current step count is greater than the first threshold, it indicates that the current step count is the interleaved step count for the last part. If the current step count is not the interleaved step count for the last part, the tensor of the step count is stored in the reverse cache queue. Referring to Figure 7, the current step count of the first compute unit can start accumulating from 1 (or 0), i.e., the current step count of the first step is 1, and the second step is 2. If the number of microbatches contained in the first subblock is 5 and the interleaved dimension is 2, the first threshold is 5. If the current step count of the first compute unit is between 1 and 5, it is not the interleaved step count for the last part, but if the current step count is between 6 and 10, it is the interleaved step count for the last part. Referring to Figure 8, if the number of microbatches in the first subblock is 5 and the interleaving dimension is 3, the first threshold is 10. If the current number of steps in the first computing unit is between 1 and 11, it is not the number of interleaved steps in the last part; if the current number of steps is between 11 and 15, it is the number of interleaved steps in the last part.

[0057] In the embodiments of this disclosure, the period from the start step number of the first subblock to the step number corresponding to the first threshold can be understood as a cache interval. Unlike the transmission interval, the cache interval allows for delayed transmission of data in the cache queue. For example, when the first compute unit computes microbatch 1, the current step number is 1, and since the current step number is less than the first threshold, the compute result of microbatch 1 can be stored in the first item (head) of the reverse cache queue. However, since the current step number is not included in the transmission interval, no data in the reverse cache queue is transmitted. When the first compute unit computes microbatch 2, the current step number is 2, and since the current step number is less than the first threshold, the compute result of microbatch 2 can be stored in the second item of the reverse cache queue. And since the current step number is included in the transmission interval, the head data in the reverse cache queue can be sent to the last compute unit, changing the second item in the reverse cache queue to the first item. Alternatively, the head data in the reverse cache queue can be sent to the last compute unit first, and then the compute result of microbatch 2 can be cached.

[0058] According to an embodiment of the present disclosure, after the first computing unit has computed a microbatch of tensors and stored them in a reverse cache queue, it is possible to determine, based on the current number of steps and transmission interval, whether to send the head tensor in the reverse cache queue to the last computing unit at the current number of steps, enabling delayed transmission of cached tensors and increasing the number of microbatches to which interleaved pipeline scheduling can be applied.

[0059] In one embodiment, determining whether to send the head tensor in the reverse cache queue to the last compute unit based on the current step number and the transmission interval is: If the current number of steps falls within the transmission interval, the scheduling value of the head tensor in the reverse cache queue is set to the first value; or, if the current number of steps falls outside the transmission interval, the scheduling value of the head tensor in the reverse cache queue is set to the second value. This includes sending the head tensor, whose scheduling value is the first value, to the last computing unit.

[0060] In the embodiments of this disclosure, the first and second values ​​are different. For example, the first value can be a non-null value, and the second value can be a null value (none). If the current step count is within the transmission interval, the scheduling value of the head tensor in the reverse cache can be set to a non-null value, for example, 1. If the current step count is not within the transmission interval, the scheduling value of the head tensor in the reverse cache queue is set to a null value, for example, none. For example, if the number of microbatches in the first subblock is 5, the interleaved dimension is 2, the current step count starts accumulating from 1, and the transmission interval is (1,6), then first, the tensor of microbatch 1 calculated and obtained at the current step count of 1 is stored in the head of the reverse cache queue. Next, since the current step count of 1 is not in the transmission interval (1,6), the scheduling value of the head tensor in the reverse cache queue is set to null. Because the scheduling value of the head tensor is null, the current step count of 1 does not transmit the head tensor. Subsequently, the current step count accumulates to 2, and the current step count of 2 is in the transmission interval (1,6), so the scheduling value of the head tensor in the reverse cache queue is set to 1. Because the scheduling value of the head tensor is 1, the head tensor (the tensor of microbatch 1) is transmitted at the current step count of 2. After transmission, the tensor of microbatch 1 can be deleted, and the tensor of microbatch 2 can be made the head tensor in the reverse cache queue.

[0061] According to embodiments of this disclosure, the timing of transmission from the first compute unit to the last compute unit can be controlled through the scheduling value of the head tensor in the reverse cache queue, thereby delaying the transmission of some microbatch tensors and increasing the number of microbatch to which interleaved pipeline scheduling can be applied.

[0062] In one embodiment, the first computing unit performs cache scheduling for the computation results of microbatches requiring delayed transmission in the reverse computing process, based on the current number of steps of the microbatch being processed, the transmission interval, and the forward cache queue. If the current number of steps is equal to or greater than the total number of steps in the first subblock, this includes setting the scheduling value of the tensor corresponding to the current number of steps to the second value.

[0063] In embodiments of this disclosure, if the current step count is greater than or equal to the total number of steps in the first subblock, the computation to be performed corresponds to other subblocks that do not require cache scheduling, and the scheduling value of the tensor corresponding to the current step count can be set to a second value, such as a null value, and there is no need to store the tensor corresponding to the current step count in a reverse cache queue or to send it with a delay.

[0064] According to the embodiments of this disclosure, other subblocks that do not require cache scheduling can be normally transmitted, thereby reducing scheduling complexity and improving computational efficiency.

[0065] In one application, the interleaved pipeline does not limit the batch size (the number of microbatches processed at once) and does not require the batch size to be divisible by the partition dimension (pp). This disclosure modifies the scheduling method for the interleaved pipeline, and the modified interleaved pipeline scheduling method supports arbitrary batch size scheduling. Figure 7 shows a scenario with a batch size of 9 and an interleaved dimension of 2.

[0066] As shown in Figure 7, in this scenario, the pipeline partitioning dimension is num_stages=4, the interleaving dimension is num_model_chunks=2, and the cumulative number of steps is acc_steps=9.

[0067] First, calculate the following microbatch scheduling parameters: the number of microbatch units in the first subblock (chunk) (first_chunk_acc), the total number of steps in the first chunk (first_chunk_steps), and the number of steps that need to be skipped (skip_steps / jump steps). An example of the calculation formula is as follows: first_chunk_acc = acc_steps % num_stages + num_stages. first_chunk_steps = first_chunk_acc * num_model_chunks. skip_steps = acc_steps % num_stages.

[0068] In one example, referring to Figure 7, first_chunk_acc=9%4+4=5, first_chunk_steps=5*2=10, and skip_steps=9%4=1. In another example, referring to Figure 8, the interleaved dimension is num_model_chunks=3, and the other parameters are the same as in Figure 7: first_chunk_acc=9%4+4=5, first_chunk_steps=5*3=15, and skip_steps=9%4=1. In yet another example, if the cumulative count is acc_steps=10, the other parameters are the same as in Figure 7: first_chunk_acc=10%4+4=6, first_chunk_steps=6*2=12, and skip_steps=10%4=2.

[0069] Next, we construct two cache queues: a forward cache queue, fwd_buffer_queue, and a reverse cache queue, bwd_buffer_queue.

[0070] Next, calculate the limiting boundaries for the first and last computing devices, denoting the left boundary value as left_id and the right boundary value as right_id. An example of the calculation formula is as follows: left_id = skip_steps; right_id = left_id + first_chunk_acc * (num_model_chunks - 1).

[0071] For example, referring to Figure 7, left_id=1 and right_id=1+5*(2-1)=6.

[0072] Subsequently, for the first computing unit (pp-stage0), the modified interleaved pipeline scheduling flow is as shown in Figure 10.

[0073] In S1001, first, it is determined whether the current step number step_id of the first computing unit is less than the total number of steps first_chunk_steps of the first subblock. If so, S1002 is executed to make the following further determination: If step_id is greater than or equal to first_chunk_steps, step S1005 is executed. S1002 can also be executed if step_id is greater than or equal to first_chunk_steps, and if the result of the determination is so, S1005 is executed.

[0074] In S1002, determine whether the first computing unit (pp-stage0) is currently executing the last part of the interleaved scheduling (or whether it is pp-stage0 of the interleaved process). If not, place the computation result corresponding to the current step number, such as a tensor, into bwd_buffer_queue. Continue executing S1003. Otherwise, execute S1005.

[0075] In S1003, the next step is to check if step_id is between left_id and right_id (if step_id starts from 0, the transmission interval includes left_id but not right_id; if step_id starts from 1, the transmission interval does not include left_id but includes right_id). If so, execute S1004; otherwise, execute S1005.

[0076] In S1004, a value is set to the tensor of bwd_buffer_queue. This value is not None. Next, execute S1005.

[0077] In S1005, set the tensor to None. Then execute S1006.

[0078] In S1006, determine whether the tensor is None. If the tensor is not None, send the tensor in the bwd_buffer_queue to the last computation unit, pp-last-stage. If it is None, it does not need to be sent.

[0079] Subsequently, the revised interleaved pipeline scheduling flow for the final computing unit (pp-last-stage) is shown in Figure 11.

[0080] In S1101, first, it is determined whether the current step number step_id of the last computing unit is less than the total step number first_chunk_steps of the first subblock. If so, S1102 is executed to make the following further determination: If step_id is greater than or equal to first_chunk_steps, step S1105 is executed. S1102 can also be executed if step_id is greater than or equal to first_chunk_steps, and if the result of the determination is so, S1105 is executed.

[0081] In S1102, determine if the last computation unit (pp-last-stage) is currently executing the last part of the interleaved scheduling (or determine if it is pp-stage0 of the interleaved scheduling). If not, place the computation result corresponding to the current step number, such as a tensor, into fwd_buffer_queue. Continue executing S1103. Otherwise, execute S1105.

[0082] In S1103, the next step is to check if step_id is between left_id and right_id (if step_id starts from 0, the transmission interval includes left_id but not right_id; if step_id starts from 1, the transmission interval does not include left_id but includes right_id). If so, execute S1104; otherwise, execute S1105.

[0083] In S1104, a value is set to the tensor of fwd_buffer_queue. This value is not None. Next, S1105 is executed.

[0084] In S1105, set the tensor to None. Then execute S1106.

[0085] In S1106, it is determined whether the tensor is None. If the tensor is not None, the tensor in the bwd_buffer_queue is sent to the first computing unit pp-stage0. If it is None, it does not need to be sent.

[0086] For the scenario with interleaved dimension num_model_chunks>2, see Figure 8, where the transmission interval can include forward or reverse computation portions between curly braces. For example, the last computation unit sends the tensor of microbatch 1 of the first forward computation to the first computation unit in step 2, sends the tensor of microbatch 2 of the first forward computation to the first computation unit in step 3, ..., and sends the tensor of microbatch 5 of the second forward computation to the first computation unit in step 11. Alternatively, the first computation unit sends the tensor of microbatch 1 of the first reverse computation to the last computation unit in step 2, sends the tensor of microbatch 2 of the first reverse computation to the last computation unit in step 3, ..., and sends the tensor of microbatch 5 of the second reverse computation to the last computation unit in step 11.

[0087] Figure 12 is a schematic diagram showing the configuration of an interleaved pipeline scheduling device according to one embodiment of the present disclosure, and the device is A parameter acquisition module 1201 for obtaining micro-batch scheduling parameters based on pipeline splitting dimension, interleave dimension, and cumulative count, A spatial determination module 1202 for determining the transmission interval of a microbatch that requires cache scheduling based on the aforementioned microbatch scheduling parameters, The system includes a cache scheduling module 1203 for performing cache scheduling on the calculation results of micro-batches requiring delayed transmission, based on the transmission interval and cache unit. Interleaved pipeline scheduling device.

[0088] In one embodiment, the pipeline partitioning dimension includes the number of computing units for performing pipeline scheduling for the model training process.

[0089] In one embodiment, the model training process is divided into multiple stages, and the interleaving dimension includes the number of stages for which one computing unit requires interleaving execution for one microbatch.

[0090] Figure 13 is a schematic diagram showing the configuration of an interleaved pipeline scheduling device according to another embodiment of the present disclosure. The device may include one or more features of the interleaved pipeline scheduling device. In one embodiment, the parameter acquisition module 1201 is A first computation submodule 1301 for obtaining the number of microbatches included in the first subblock based on the pipeline partitioning dimension and the cumulative count, A second computation submodule 1302 for obtaining the total number of steps in the first subblock based on the number of microbatches contained in the first subblock and the interleaved dimension, It includes a third computation submodule 1303 for obtaining the number of jump steps based on the pipeline division dimension and the cumulative number of jumps.

[0091] In one embodiment, as shown in Figure 13, the spatial determination module 1202 is:

[0092] A first boundary determination submodule for obtaining a first boundary of the transmission section based on the number of jump steps, wherein the first boundary is used to determine the starting number of steps for which cached data must be transmitted, and the first boundary determination submodule 1304 A second boundary determination submodule 1305 for obtaining a second boundary of the transmission interval based on the first boundary, the number of microbatches contained in the first subblock, and the interleave dimension, wherein the second boundary is used to determine the number of steps for which transmission of cached data should be stopped.

[0093] In one embodiment, the cache scheduling module 1203 is further used to cache schedule the computation results of microbatches that require delayed transmission in the forward computation process by the last compute unit, based on the current number of steps of the microbatch being processed by the last compute unit, the transmission interval, and the forward cache queue.

[0094] In one embodiment, as shown in Figure 13, the cache scheduling module 1203 is: If the current number of steps in the microbatch being processed by the last computing unit is less than the total number of steps in the first subblock, and the current number of steps is not the number of interleaved steps in the last part, the first storage submodule 1306 stores the tensor corresponding to the current number of steps in the forward cache queue, The system includes a first transmit submodule 1307 for determining whether to send the head tensor in the forward cache queue to the first compute unit based on the current step number and the transmit interval.

[0095] In one embodiment, as shown in Figure 13, the cache scheduling module 1203 further, The system includes a first setting submodule 1308 for setting the scheduling value of the head tensor in the forward cache queue to a first value if the current number of steps is within the transmission interval, or for setting the scheduling value of the head tensor in the forward cache queue to a second value if the current number of steps is not within the transmission interval. The first transmission submodule 1307 is further used to transmit the head tensor, whose scheduling value is the first value, to the first computing unit.

[0096] In one embodiment, as shown in Figure 13, the cache scheduling module 1203 further, If the current number of steps is greater than or equal to the total number of steps in the first subblock, it includes a second setting submodule 1309 for setting the scheduling value of the tensor corresponding to the current number of steps to a second value.

[0097] In one embodiment, the cache scheduling module 1203 is further used to cache schedule the computation results of microbatches that require delayed transmission in the reverse computation process by the first compute unit, based on the current number of steps of the microbatch being processed by the first compute unit, the transmission interval, and the reverse cache queue.

[0098] In one embodiment, the cache scheduling module 1203 further, If the current number of steps in the microbatch being processed by the first computing unit is less than the total number of steps in the first subblock, and the current number of steps is not the number of interleaved steps in the last part, a second storage submodule 1310 for storing the tensor corresponding to the current number of steps in the reverse cache queue, The system includes a second transmit submodule 1311 for determining whether to send the head tensor in the reverse cache queue to the last compute unit based on the current step number and the transmit interval.

[0099] In one embodiment, as shown in Figure 13, the cache scheduling module 1203 further, The system includes a third setting submodule 1312 for setting the scheduling value of the head tensor in the reverse cache queue to a first value if the current number of steps is within the transmission interval, or for setting the scheduling value of the head tensor in the reverse cache queue to a second value if the current number of steps is not within the transmission interval. The second transmission submodule 1311 is further used to transmit the head tensor, whose scheduling value is the first value, to the last computing unit.

[0100] In one embodiment, as shown in 13, the cache scheduling module 1203 further, If the current number of steps is greater than or equal to the total number of steps in the first subblock, it includes a fourth setting submodule 1313 for setting the scheduling value of the tensor corresponding to the current number of steps to a second value.

[0101] A description of the specific functions and examples of each module and submodule of the apparatus of the embodiments of this disclosure can be found in the relevant descriptions of the corresponding steps in the above-described method embodiments and will not be repeated here.

[0102] In the technical solutions disclosed herein, the acquisition, storage, and use of relevant user personal information will comply with the provisions of relevant laws and regulations and will not violate public order and morals.

[0103] According to embodiments of the present disclosure, the present disclosure further provides electronic devices, non-temporary computer-readable storage media, and program products.

[0104] Figure 14 is a block diagram of an electronic device 1400 according to an embodiment of the present disclosure. An electronic device refers to any form of digital computer, such as a laptop computer, desktop computer, workstation, personal digital assistant, server, blade server, mainframe computer, and other suitable computers. An electronic device further refers to any form of mobile device, such as a personal digital assistant, cellular phone, intelligent phone, wearable device, and other similar computer equipment. The components, their connections, and functions described in this disclosure are illustrative and do not limit the realization of anything described or specified in this disclosure.

[0105] As shown in Figure 14, device 1400 includes a computing unit 1401 capable of performing various appropriate operations and processes based on computer program instructions stored in read-only memory (ROM) 1402, or computer program instructions loaded from storage unit 1408 into random access memory (RAM) 1403. RAM 1403 can further store various programs and data necessary for the operation of device 1400. The computing unit 1401, ROM 1402, and RAM 1403 are connected to each other via bus 1404. An input / output (I / O) interface 1405 is also connected to bus 1404.

[0106] Multiple components in device 1400 are connected to an I / O interface 1405, which includes an input unit 1406 such as a keyboard and mouse, an output unit 1407 such as various displays and speakers, a storage unit 1408 such as a magnetic disk or optical disk, and a communication unit 1409 such as a network card, modem, or wireless communication transceiver. The communication unit 1409 allows device 1400 to exchange information / data with other devices via computer networks such as the Internet and / or various carrier networks.

[0107] The computing unit 1401 may be a variety of general-purpose and / or dedicated processing components having processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, a computing unit that executes various machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1401 performs each of the methods and processes described above, for example, the interleaved pipeline scheduling method. For example, in some embodiments, the interleaved pipeline scheduling method can be implemented as a computer software program tangibly contained in a machine-readable medium such as a storage unit 1408. In some embodiments, part or all of the computer program can be loaded and / or installed into device 1400 via ROM 1402 and / or communication unit 1409. When the computer program is loaded into RAM 1403 and executed by the computing unit 1401, one or more steps of the interleaved pipeline scheduling method described above can be performed. In addition, in other embodiments, the computing unit 1401 may be configured to perform the interleaved pipeline scheduling method by any other suitable method (e.g., firmware).

[0108] Various embodiments of the systems or technologies described in this disclosure can be implemented by digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standards (ASSPs), systems-on-a-chip (SOCs), complex-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. Each of these embodiments may be implemented by one or more computer programs that run and / or interpret on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0109] Program code for performing the methods of this disclosure can be written in any combination of one or more programming languages. These program codes are provided to a processor or controller of a general-purpose computer, a dedicated computer, or other programming data processing device, so that when the program code is executed by the processor or controller, it can perform the functions / operations defined in the flowcharts and / or block diagrams. The program code may run entirely in a mainscan, partially in a mainscan, partially as an independent soft encapsulation and partially in a remote mainscan, or entirely in a remote mainscan or server.

[0110] In this disclosure, machine-readable media may be tangible media containing or storing programs used by or in conjunction with instruction execution systems, devices, or equipment. Machine-readable media may be machine-readable signal media or machine-readable storage media. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any suitable combination of the contents described above. Further specific examples of machine-readable storage media include one or more wired electrical connections, portable computer disk cartridges, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any combination of the contents described above.

[0111] To provide user interaction, a computer may implement the systems and technologies described herein, which may include a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor), a keyboard and pointing device for the user to provide input to the computer (e.g., a mouse or trackball). Other types of devices may also be used to provide user interaction; for example, the feedback provided to the user may be any form of sensor feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and input from the user may be accepted in any form (e.g., acoustic input, voice input, haptic input).

[0112] The systems and technologies described herein can be implemented in computing systems that include background components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include front-end components (e.g., user computers having a graphics user interface or network browser, through which users can interact with embodiments of the systems and technologies described herein), or in any combination of such background components, middleware components, or front-end components. Components of the system can be connected to one another via digital data communication in any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0113] A computer system can include a client and a server. Typically, the client and server are geographically separated and interact via a communication network. The client-server relationship is created by a computer program that operates on the corresponding computer. The server may be a cloud server, a server in a distributed system, or a server incorporating blockchain technology, etc.

[0114] It should be understood that steps can be newly ranked, added, or deleted using the various forms of flows shown above. For example, each step described in this disclosure may be executed in parallel, sequentially, or in a different order. This disclosure is not limited to this, as long as the technical solutions disclosed herein can achieve the desired results.

[0115] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, subcombinations, and substitutions are possible due to design considerations and other factors. Any changes, equivalent substitutions, and improvements within the gist and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An interleaved pipeline scheduling method, Based on the pipeline splitting dimension, interleave dimension, and cumulative count, micro-batch scheduling parameters are obtained, Based on the aforementioned micro-batch scheduling parameters, the transmission interval of the micro-batch requiring cache scheduling is determined, This includes performing cache scheduling for the calculation results of micro-batches requiring delayed transmission, based on the transmission interval and cache unit. Interleaved pipeline scheduling method.

2. The pipeline partitioning dimension includes the number of computing units for performing pipeline scheduling for the model training process. The interleaved pipeline scheduling method according to claim 1.

3. The model training process is divided into multiple stages, and the interleaving dimension includes the number of stages for which one computing unit requires interleaving execution for one microbatch. The interleaved pipeline scheduling method according to claim 1 or 2.

4. Based on the pipeline splitting dimension, interleave dimension, and cumulative count, obtaining micro-batch scheduling parameters is possible. Based on the pipeline partitioning dimension and the cumulative count, the number of microbatches included in the first subblock is obtained, The total number of steps in the first subblock is obtained based on the number of microbatches included in the first subblock and the interleaved dimension. This includes obtaining the number of jump steps based on the pipeline division dimension and the cumulative number of times, An interleaved pipeline scheduling method according to any one of claims 1 to 3.

5. Based on the aforementioned micro-batch scheduling parameters, determining the transmission interval of a micro-batch that requires cache scheduling is: The first boundary of the transmission section is obtained based on the number of jump steps, and the first boundary is used to determine the starting number of steps for which cached data needs to be transmitted. A second boundary of the transmission interval is obtained based on the first boundary, the number of microbatches included in the first subblock, and the interleaved dimension, wherein the second boundary is used to determine the number of steps for which transmission of cached data should be stopped. The interleaved pipeline scheduling method according to claim 4.

6. Based on the transmission interval and cache unit, cache scheduling is performed for the calculation results of the micro-batch requiring delayed transmission. The last computing unit performs cache scheduling for the calculation results of microbatches that require delayed transmission in the forward computing process, based on the current number of steps of the microbatch being processed by the last computing unit, the transmission interval, and the forward cache queue. The interleaved pipeline scheduling method according to claim 4 or 5.

7. Based on the current number of steps in the microbatch being processed by the last computing unit, the transmission interval, and the forward cache queue, the last computing unit performs cache scheduling for the calculation results of microbatches that require delayed transmission in the forward computing process. If the current number of steps in the microbatch being processed by the last computing unit is less than the total number of steps in the first subblock, and the current number of steps is not the number of interleaved steps in the last part, then the tensor corresponding to the current number of steps is stored in the forward cache queue. The process includes determining whether to send the head tensor in the forward cache queue to the first compute unit based on the current step number and the transmission interval, The interleaved pipeline scheduling method according to claim 6.

8. Based on the current step count and the transmission interval, determining whether to send the head tensor in the forward cache queue to the first compute unit is: If the current number of steps falls within the transmission interval, the scheduling value of the head tensor in the forward cache queue is set to a first value; or, if the current number of steps falls outside the transmission interval, the scheduling value of the head tensor in the forward cache queue is set to a second value. This includes transmitting the head tensor, whose scheduling value is a first value, to the first computing unit, The interleaved pipeline scheduling method according to claim 7.

9. Based on the current number of steps in the microbatch being processed by the last computing unit, the transmission interval, and the forward cache queue, the last computing unit performs cache scheduling for the calculation results of microbatches that require delayed transmission in the forward computing process. If the current number of steps is equal to or greater than the total number of steps in the first subblock, the scheduling value of the tensor corresponding to the current number of steps is set to the second value. An interleaved pipeline scheduling method according to any one of claims 6 to 8.

10. Based on the transmission interval and cache unit, cache scheduling is performed for the calculation results of the micro-batch requiring delayed transmission. The first computing unit performs cache scheduling for the calculation results of microbatches that require delayed transmission in the reverse computing process, based on the current number of steps of the microbatch being processed by the first computing unit, the transmission interval, and the reverse cache queue. The interleaved pipeline scheduling method according to claim 4 or 5.

11. Based on the current number of steps in the microbatch being processed by the first computing unit, the transmission interval, and the reverse cache queue, the first computing unit performs cache scheduling for the calculation results of microbatches that require delayed transmission in the reverse computing process. If the current number of steps in the microbatch being processed by the first computing unit is less than the total number of steps in the first subblock, and the current number of steps is not the number of interleaved steps in the last part, then the tensor corresponding to the current number of steps is stored in the reverse cache queue. The process includes determining whether to send the head tensor in the reverse cache queue to the last compute unit based on the current step number and the transmission interval, The interleaved pipeline scheduling method according to claim 10.

12. Based on the current step count and the transmission interval, determining whether to send the head tensor in the reverse cache queue to the last compute unit is: If the current number of steps falls within the transmission interval, the scheduling value of the head tensor in the reverse cache queue is set to a first value; or, if the current number of steps falls outside the transmission interval, the scheduling value of the head tensor in the reverse cache queue is set to a second value. This includes transmitting the head tensor, whose scheduling value is a first value, to the last computing unit, The interleaved pipeline scheduling method according to claim 11.

13. Based on the current number of steps in the microbatch being processed by the first computing unit, the transmission interval, and the reverse cache queue, the first computing unit performs cache scheduling for the calculation results of microbatches that require delayed transmission in the reverse computing process. If the current number of steps is equal to or greater than the total number of steps in the first subblock, the scheduling value of the tensor corresponding to the current number of steps is set to the second value. An interleaved pipeline scheduling method according to any one of claims 10 to 12.

14. An interleaved pipeline scheduling device, A parameter acquisition module for obtaining micro-batch scheduling parameters based on pipeline splitting dimension, interleave dimension, and cumulative count, A spatial determination module for determining the transmission interval of a microbatch that requires cache scheduling based on the aforementioned microbatch scheduling parameters, The system includes a cache scheduling module for performing cache scheduling on the calculation results of micro-batches requiring delayed transmission, based on the transmission interval and cache unit. Interleaved pipeline scheduling device.

15. The pipeline partitioning dimension includes the number of computing units for performing pipeline scheduling for the model training process. The interleaved pipeline scheduling device according to claim 14.

16. The model training process is divided into multiple stages, and the interleaving dimension includes the number of stages for which one computing unit requires interleaving execution for one microbatch. The interleaved pipeline scheduling device according to claim 14 or 15.

17. The parameter acquisition module described above is: A first computation submodule for obtaining the number of microbatches included in the first subblock based on the pipeline partitioning dimension and the cumulative count, A second computation submodule for obtaining the total number of steps in the first subblock based on the number of microbatches included in the first subblock and the interleaving dimension, A third computation submodule for obtaining the number of jump steps based on the pipeline partitioning dimension and the cumulative number of times, An interleaved pipeline scheduling device according to any one of claims 14 to 16.

18. The aforementioned spatial decision module is A first boundary determination submodule for obtaining a first boundary of the transmission section based on the number of jump steps, wherein the first boundary is used to determine the starting number of steps for which cached data must be transmitted, A second boundary determination submodule for obtaining a second boundary of the transmission interval based on the first boundary, the number of microbatches included in the first subblock, and the interleaved dimension, wherein the second boundary is used to determine the number of steps for which transmission of cached data should be stopped, The interleaved pipeline scheduling device according to claim 17.

19. The cache scheduling module further, The last computing unit uses the current number of steps of the microbatch being processed, the transmission interval, and the forward cache queue to perform cache scheduling for the calculation results of microbatches that require delayed transmission in the forward computing process. The interleaved pipeline scheduling device according to claim 17 or 18.

20. The aforementioned cache scheduling module is If the current number of steps in the microbatch being processed by the last computing unit is less than the total number of steps in the first subblock, and the current number of steps is not the number of interleaved steps in the last part, then a first storage submodule for storing the tensor corresponding to the current number of steps in the forward cache queue, A first transmit submodule for determining whether to transmit the head tensor in the forward cache queue to the first compute unit based on the current step number and the transmit interval, The interleaved pipeline scheduling device according to claim 19.

21. The cache scheduling module further, The system includes a first setting submodule for setting the scheduling value of the head tensor in the forward cache queue to a first value if the current number of steps falls within the transmission interval, or for setting the scheduling value of the head tensor in the forward cache queue to a second value if the current number of steps falls outside the transmission interval. The first transmission submodule is further used to transmit the head tensor, whose scheduling value is a first value, to the first computing unit. The interleaved pipeline scheduling device according to claim 20.

22. The cache scheduling module further, If the current number of steps is equal to or greater than the total number of steps in the first subblock, a second setting submodule is included for setting the scheduling value of the tensor corresponding to the current number of steps to a second value. An interleaved pipeline scheduling device according to any one of claims 19 to 21.

23. The cache scheduling module further, The first computing unit uses the current number of steps of the microbatch being processed, the transmission interval, and the reverse cache queue to perform cache scheduling for the calculation results of microbatches that require delayed transmission in the reverse computing process. The interleaved pipeline scheduling device according to claim 17 or 18.

24. The cache scheduling module further, If the current number of steps in the microbatch being processed by the first computing unit is less than the total number of steps in the first subblock, and the current number of steps is not the number of interleaved steps in the last part, a second storage submodule for storing the tensor corresponding to the current number of steps in the reverse cache queue, A second transmit submodule for determining whether to transmit the head tensor in the reverse cache queue to the last compute unit based on the current step number and the transmit interval, The interleaved pipeline scheduling device according to claim 23.

25. The cache scheduling module further, The system includes a third setting submodule for setting the scheduling value of the head tensor in the reverse cache queue to a first value if the current number of steps falls within the transmission interval, or for setting the scheduling value of the head tensor in the reverse cache queue to a second value if the current number of steps falls outside the transmission interval. The second transmission submodule is further used to transmit the head tensor, whose scheduling value is a first value, to the last computing unit. The interleaved pipeline scheduling device according to claim 24.

26. The cache scheduling module further, If the current number of steps is equal to or greater than the total number of steps in the first subblock, a fourth setting submodule is included for setting the scheduling value of the tensor corresponding to the current number of steps to a second value. An interleaved pipeline scheduling device according to any one of claims 23 to 25.

27. At least one processor, The system comprises at least one processor and a memory that is communicated with by it, The memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor causes the at least one processor to perform the method according to any one of claims 1 to 13. Electronic devices.

28. A non-temporary computer-readable storage medium storing computer instructions that cause a computer to perform the method described in any one of claims 1 to 13.

29. A program product comprising a program that, when executed by a processor in a computer, implements the method described in any one of claims 1 to 13.

Images