Staggered assembly line parallel training method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention discloses a staggered assembly line parallel training method and device, equipment, a storage medium and a program product, and belongs to the technical field of artificial intelligence. The method comprises the steps that based on the number of calculation units, the number of blocks and the total number of layers of a large model, the number of layers contained in each block is determined, and the number of layers contained in each block is configurable; based on the number of calculation units, the number of blocks and the total number of layers of the large model, determining that the number of layers contained in each block comprises one of the following layers: based on the number of calculation units, the number of blocks and the total number of layers, determining the number of layers contained in each block by taking bubble time as an index; based on the calculation unit number, the block number and the total layer number, determining the layer number contained in each block by taking the video memory consumption as an index; based on the calculation unit number, the block number and the total layer number, determining the layer number contained in each block by taking the bubble time and the video memory consumption as indexes; and on the basis of the number of layers contained in each block, performing staggered assembly line parallel training on the large model. The training efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence (AI) technology, and more specifically, to an interleaved pipeline parallel training method, device, equipment, storage medium and program product. Background Art

[0002] In the field of deep learning, computing units (such as GPUs or GPGPUs) are usually used to perform model training. The memory capacity of computing units is generally limited. For large model training scenarios, the parameter scale of large models is usually huge (for example, the number of parameters is between one billion and one trillion). Pipeline Parallelism (PP) refers to dividing the model according to the number of layers, deploying different layers of the model to different computing units for pipeline scheduling, and each computing unit is called a different PP stage.

[0003] Pipeline parallel training usually includes non-interleaved and interleaved. Currently, interleaved pipeline parallel training can only be performed when the number of layers of computing units is divisible by the number of chunks, which is not conducive to improving training efficiency. Summary of the invention

[0004] The present invention provides an interleaved pipeline parallel training method, device, equipment, storage medium and program product, which are helpful to improve training efficiency.

[0005] The technical solution of the embodiment of the present invention is as follows: A staggered pipeline parallel training method, comprising: Based on the number of computing units, the number of blocks and the total number of layers of the large model, the number of layers contained in each block is determined, wherein the number of layers contained in each block is configurable; the determination of the number of layers contained in each block based on the number of computing units, the number of blocks and the total number of layers of the large model includes one of the following: based on the number of computing units, the number of blocks and the total number of layers, the number of layers contained in each block is determined with bubble time as an indicator; based on the number of computing units, the number of blocks and the total number of layers, the number of layers contained in each block is determined with video memory consumption as an indicator; based on the number of computing units, the number of blocks and the total number of layers, the number of layers contained in each block is determined with bubble time and video memory consumption as indicators; Based on the number of layers contained in each block, staggered pipeline parallel training is performed on the large model.

[0006] In one embodiment, the determining the number of layers contained in each block based on the number of computing units, the number of blocks and the total number of layers and taking the bubble time as an indicator comprises: Based on the number of computing units and the number of blocks, determining a plurality of candidate solutions for allocating the total number of layers to each block, each candidate solution including the number of layers included in each block; Determine the bubble time for each candidate solution; Selecting a candidate solution with the shortest bubble time from the plurality of candidate solutions; The number of layers included in each block is determined based on the candidate solution with the shortest bubble time.

[0007] In one embodiment, determining the bubble time of each candidate solution includes: Determine the forward time for each candidate solution to perform forward computation on a unit micro-batch for each block; Determine a maximum forward time from the multiple forward times of the multiple blocks of each candidate solution; Determine the reverse time for each block of each candidate solution to perform the reverse calculation on the unit micro-batch; Determine a maximum reverse time from multiple reverse times of multiple blocks of each candidate solution; The bubble time of each candidate solution is determined based on the maximum forward time, the maximum reverse time and the number of computing units.

[0008] In one embodiment, the determining the number of layers included in each block based on the number of computing units, the number of blocks, and the total number of layers and taking video memory consumption as an indicator includes: Based on the number of computing units and the number of blocks, determining a plurality of candidate solutions for allocating the total number of layers to each block, each candidate solution including the number of layers included in each block; Determine the video memory consumption of the computing unit that first performs the calculation in each candidate solution; Selecting the candidate solution with the smallest video memory consumption from the multiple candidate solutions; Based on the candidate solution with the minimum video memory consumption, the number of layers included in each block is determined.

[0009] In one embodiment, determining the video memory consumption of the computing unit that first performs the calculation in each candidate solution includes: Determine a first activation value for all blocks of the computing unit that first performs the calculation on the unit micro-batch in each candidate solution; Determine a second activation value of a first block of a computing unit that first performs calculations in each candidate solution for performing calculations on a unit micro-batch; Based on the first activation value, the second activation value, and the number of computing units, a video memory consumption of a computing unit that first performs calculations in each candidate solution is determined.

[0010] In one embodiment, the determining the number of layers included in each block based on the number of computing units, the number of blocks, and the total number of layers, taking bubble time and video memory consumption as indicators, comprises: Based on the number of computing units and the number of blocks, determining a plurality of candidate solutions for allocating the total number of layers to each block, each candidate solution including the number of layers included in each block; Determine the video memory consumption of the computing unit that first performs the calculation in each candidate solution; Selecting at least one candidate solution whose video memory consumption is lower than a first predetermined threshold from the multiple candidate solutions; determining a bubble time for the at least one candidate solution; Selecting a candidate solution with the shortest bubble time from the at least one candidate solution; The number of layers included in each block is determined based on the candidate solution with the shortest bubble time.

[0011] In one embodiment, the determining the number of layers included in each block based on the number of computing units, the number of blocks, and the total number of layers, taking bubble time and video memory consumption as indicators, comprises: Based on the number of computing units and the number of blocks, determining a plurality of candidate solutions for allocating the total number of layers to each block, each candidate solution including the number of layers included in each block; Determine the bubble time for each candidate solution; Selecting at least one candidate solution having a bubble time lower than a second predetermined threshold from the plurality of candidate solutions; Determine the video memory consumption of the computing unit that first performs the calculation in the at least one candidate solution; Selecting a candidate solution with the smallest video memory consumption from the at least one candidate solution; Based on the candidate solution with the minimum video memory consumption, the number of layers included in each block is determined.

[0012] In one embodiment, the determining the number of layers included in each block based on the number of computing units, the number of blocks, and the total number of layers, taking bubble time and video memory consumption as indicators, comprises: Based on the number of computing units and the number of blocks, determining a plurality of candidate solutions for allocating the total number of layers to each block, each candidate solution including the number of layers included in each block; Determine the bubble time of each candidate solution and the video memory consumption of the computing unit that first performs the calculation in each candidate solution; Selecting at least one candidate solution from the plurality of candidate solutions, the video memory consumption of which is lower than a first predetermined threshold and the bubble time of which is lower than a second predetermined threshold; Selecting a candidate solution with the smallest video memory consumption or a candidate solution with the smallest bubble time from the at least one candidate solution; Based on the selected candidate solution, the number of layers included in each block is determined.

[0013] In one embodiment, before determining the number of layers contained in each block based on the number of computing units, the number of blocks, and the total number of layers of the large model, the method further includes: Determining the number of layers of each computing unit based on the total number of layers and the number of computing units; The method of determining the number of layers contained in each block based on the number of computing units, the number of blocks, and the total number of layers of the large model includes: When the number of layers of each computing unit is not divisible by the number of blocks, the number of layers included in at least two blocks is different; When the number of layers of each computing unit is divisible by the number of blocks, the number of layers contained in each block is the same.

[0014] An interleaved pipeline parallel training device, applied to any of the above interleaved pipeline parallel training methods, comprising: A determination module, used for determining the number of layers contained in each block based on the number of computing units, the number of blocks and the total number of layers of the large model, wherein the number of layers contained in each block is configurable; The training module is used to perform interleaved pipeline parallel training on the large model based on the number of layers contained in each block; the determination module is used to perform one of the following: Based on the number of computing units, the number of blocks and the total number of layers, the number of layers included in each block is determined with bubble time as an indicator; Based on the number of computing units, the number of blocks and the total number of layers, the number of layers included in each block is determined with video memory consumption as an indicator; Based on the number of computing units, the number of blocks and the total number of layers, the number of layers included in each block is determined with bubble time and video memory consumption as indicators.

[0015] In one embodiment, the determination module is used to determine multiple candidate schemes for allocating the total number of layers to each block based on the number of computing units and the number of blocks, each candidate scheme containing the number of layers contained in each block; determine the bubble time of each candidate scheme; select the candidate scheme with the shortest bubble time from the multiple candidate schemes; and determine the number of layers contained in each block based on the candidate scheme with the shortest bubble time.

[0016] In one embodiment, the determination module is used to determine multiple candidate schemes for allocating the total number of layers to each block based on the number of computing units and the number of blocks, each candidate scheme containing the number of layers contained in each block; determine the video memory consumption of the computing unit that first performs calculations in each candidate scheme; select the candidate scheme with the minimum video memory consumption from the multiple candidate schemes; and determine the number of layers contained in each block based on the candidate scheme with the minimum video memory consumption.

[0017] In one embodiment, the determining module is used to perform one of the following: Based on the number of computing units and the number of blocks, a plurality of candidate solutions for allocating the total number of layers to the respective blocks are determined, each candidate solution including the number of layers included in each block; a video memory consumption of a computing unit that first performs calculations in each candidate solution is determined; at least one candidate solution whose video memory consumption is lower than a first predetermined threshold is selected from the plurality of candidate solutions; a bubble time of the at least one candidate solution is determined; a candidate solution with the least bubble time is selected from the at least one candidate solution; based on the candidate solution with the least bubble time, the number of layers included in each block is determined; Based on the number of computing units and the number of blocks, multiple candidate solutions for allocating the total number of layers to each block are determined, each candidate solution includes the number of layers included in each block; the bubble time of each candidate solution is determined; at least one candidate solution whose bubble time is lower than a second predetermined threshold is selected from the multiple candidate solutions; the video memory consumption of the computing unit that first performs the calculation in the at least one candidate solution is determined; the candidate solution with the smallest video memory consumption is selected from the at least one candidate solution; based on the candidate solution with the smallest video memory consumption, the number of layers included in each block is determined; based on the number of computing units and the number of blocks, multiple candidate solutions for allocating the total number of layers to each block are determined, each candidate solution includes the number of layers included in each block; the bubble time of each candidate solution and the video memory consumption of the computing unit that first performs the calculation in each candidate solution are determined; at least one candidate solution whose video memory consumption is lower than the first predetermined threshold and the bubble time is lower than the second predetermined threshold is selected from the multiple candidate solutions; the candidate solution with the smallest video memory consumption or the candidate solution with the least bubble time is selected from the at least one candidate solution; based on the selected candidate solution, the number of layers included in each block is determined.

[0018] An electronic device, comprising: Memory; processor; The memory stores an application program executable by the processor, which is used to enable the processor to execute any of the above-mentioned interleaved pipeline parallel training methods.

[0019] A computer-readable storage medium having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by a processor, the processor is caused to execute any of the above-described interleaved pipeline parallel training methods.

[0020] A program product comprises a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned interleaved pipeline parallel training methods.

[0021] It can be seen from the above technical scheme that in the implementation mode of the present invention, the number of layers contained in each block is determined based on the number of computing units, the number of blocks and the total number of layers of the large model, wherein the number of layers contained in each block is configurable, wherein the number of layers contained in each block is determined based on the number of computing units, the number of blocks and the total number of layers of the large model, including one of the following: based on the number of computing units, the number of blocks and the total number of layers, the number of layers contained in each block is determined with the bubble time as an indicator; based on the number of computing units, the number of blocks and the total number of layers, the number of layers contained in each block is determined with the video memory consumption as an indicator; based on the number of computing units, the number of blocks and the total number of layers, the number of layers contained in each block is determined with the bubble time and the video memory consumption as indicators; based on the number of layers contained in each block, the large model is performed with interleaved pipeline parallel training. It can be seen that the constraint that the number of layers responsible for each computing unit must be divisible by the number of blocks is broken, and an interleaved pipeline parallel training process that supports both equal and unequal division is realized, which improves the training efficiency and expands the applicable scenarios of interleaved pipeline parallel training. In addition, the number of layers included in the block can be allocated based on the bubble time or the video memory consumption, and a good compromise can be achieved between the bubble time and the video memory consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is an exemplary schematic diagram of a non-interleaved 1F1B (One Forward pass followed by OneBackward pass) pipeline parallel training process according to the related art.

[0023] Figure 2 FIG. 4 is an exemplary schematic diagram of an interleaved pipeline parallel training process according to the related art.

[0024] Figure 3 4 is an exemplary flow chart of an interleaved pipeline parallel training method according to an embodiment of the present invention.

[0025] Figure 4 is an exemplary schematic diagram of an interleaved pipeline parallel training process according to an embodiment of the present invention.

[0026] Figure 5 4 is an exemplary structural diagram of an interleaved pipeline parallel training device according to an embodiment of the present invention.

[0027] Figure 6 is an exemplary structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0029] For the sake of brevity and intuitiveness in description, the scheme of the present invention is explained below by describing several representative implementations. A large number of details in the implementations are only used to help understand the scheme of the present invention. However, it is obvious that the technical scheme of the present invention may not be limited to these details when implemented. In order to avoid unnecessarily obscuring the scheme of the present invention, some implementations are not described in detail, but only a framework is given. Hereinafter, "including" means "including but not limited to", and "according to..." means "at least according to..., but not limited to only according to...". Due to the language habits of Chinese, when the number of a component is not specifically specified below, it means that the component can be one or more, or can be understood as at least one.

[0030] In pipeline parallel training (usually 1F1B), each layer of the large model is assigned to its own computing unit, and each training batch is divided into multiple micro batches. Each micro batch performs forward and backward propagation in sequence between different computing units, which can maximize the utilization of computing equipment and improve training efficiency.

[0031] Figure 1 The figure is an exemplary schematic diagram of the non-interleaved 1F1B pipeline parallel training process according to the related art. The core idea of ​​the 1F1B strategy is to perform a backward pass immediately after a micro-batch forward propagation of a large model, thereby reducing memory usage and parallel bubbles, improving training efficiency and throughput. This strategy is particularly effective when processing long sequences, and can significantly improve the efficiency and stability of large model training. Under this scheduling mechanism, the computing device of each stage starts processing and generates output immediately after receiving the output of the previous stage, and then passes it to the computing device of the next stage. However, this scheduling mechanism may lead to inefficient memory usage or the generation of pipeline bubbles (i.e., idle time).

[0032] exist Figure 1 In the description, four computing units A~D are taken as an example. Figure 1As shown, it is assumed that the number of micro-batches that can be processed in a non-interleaved scheduling is 8, that is, the cumulative number of times is 8. Computing units A~D perform their respective computing processes in parallel. Among them: For computing unit A, the following operations are performed in order: (1) Forward operation of micro-batch numbers 1 to 4 ( Figure 1 The dark blue boxes with numbers 1 to 4 are marked in the figure); (2) Waiting (i.e., bubble time, Figure 1 Marked with a white box); (3) Reverse operation of micro-batch number 1 ( Figure 1 (4) Forward operation of micro-batch number 5 ( Figure 1 (5) Reverse operation of micro-batch number 2 ( Figure 1 are marked with a light green box with the number 2 in the middle)… and so on.

[0033] For computing unit B, the following are executed in sequence: (1) Waiting; (2) Forward operation of micro-batch numbers 1 to 4; (3) Waiting; (4) Reverse operation of micro-batch number 1; (5) Waiting; (6) Reverse operation of micro-batch number 2, and so on.

[0034] For computing unit C, the following are executed in sequence: (1) Waiting; (2) Forward operation of micro-batch numbers 1 to 2; (3) Waiting; (4) Reverse operation of micro-batch number 1; (5) Forward operation of micro-batch number 3; (6) Reverse operation of micro-batch number 2... and so on.

[0035] For computing unit D, the following are executed in sequence: (1) waiting; (2) forward operation of micro-batch number 1; (3) reverse operation of micro-batch number 1; (4) forward operation of micro-batch number 2; (5) reverse operation of micro-batch number 2; (6) forward operation of micro-batch number 3, and so on.

[0036] visible, Figure 1 There are a lot of bubble times (i.e., a lot of white boxes) in the pipeline parallelism.

[0037] Currently, staggered 1F1B pipeline scheduling is used to try to solve the technical problem of long bubble time. Multiple chunks are allocated to computing units to reduce idle time and improve overall throughput. In staggered 1F1B pipeline scheduling, each computing unit can calculate several layers in the model.

[0038] Figure 2 FIG. 1 is an exemplary schematic diagram of an interleaved pipeline parallel training process according to the related art. In the interleaved scheduling, each computing unit is assigned multiple blocks (in Figure 2In the example, there are two blocks), allowing the forward and reverse interleaving execution of multiple blocks in different stages. For example, for interleaved 1F1B, a finer-grained segmentation is performed on the basis of non-interleaved 1F1B, and all layers originally handled by each computing unit are further segmented according to the number of blocks. Interleaved calls between multiple blocks can reduce bubble time.

[0039] exist Figure 2 In the example, four computing units A~D are used for description. Assuming that the number of micro-batches that can be processed in one interleaved scheduling is 8 (that is, the number of accumulations is 8), and the total number of layers of the large model is 40, then each computing unit is responsible for calculating 40 / 4=10 layers. Assuming that the number of blocks is 2, divided into block 1 and block 2, the number of layers in each block is 10 / 2=5. Assuming that block 1 contains layers 1 to 5, and block 2 includes layers 6 to 10.

[0040] Taking computing unit A as an example, the following are executed in order: (1) Forward operation of block 1 (including layers 1 to 5) of micro-batch numbers 1 to 4 ( Figure 2 (2) Forward operation of block 2 (including layers 6 to 10) with mini-batch numbers 1 to 4 ( Figure 2 (3) Forward operation of block 1 (including layers 1 to 5) with mini-batch numbers 5 to 6 ( Figure 2 (4) Waiting (i.e., bubble time, Figure 2 (5) Forward operation of block 1 (including layers 1 to 5) of micro-batch number 7 ( Figure 2 (6) Reverse operation of block 2 (including layers 6 to 10) of micro-batch number 1 ( Figure 2 (7) Forward operation of block 1 (including layers 1 to 5) of micro-batch number 8 ( Figure 2 (8) Reverse operation of block 2 (including layers 6 to 10) of micro-batch number 2 ( Figure 2 (9) Forward operation of block 2 (including layers 1 to 5) of micro-batch number 5 ( Figure 2 (10) Reverse operation of block 2 (including layers 6 to 10) of micro-batch number 3 ( Figure 2 (11) Forward operation of block 2 (including layers 6 to 10) of micro-batch number 6 ( Figure 2(12) Reverse operation of block 2 (including layers 6 to 10) of micro-batch number 4 ( Figure 2 (13) Forward operation of block 2 (including layers 6 to 10) of micro-batch number 7 ( Figure 2 (13) Reverse operation of block 1 (including layers 1 to 5) of micro-batch number 1 ( Figure 2 dark green box with the number 1 in the figure); ... and so on.

[0041] Similarly, computational units B~D are assigned 2 blocks, allowing interleaved execution of forward and reverse between different stages.

[0042] It can be seen that compared Figure 1 , Figure 2 The waiting time (i.e., bubbles) in the pipeline parallelism is significantly reduced. Figure 2 In the related technology, for example, only when the number of layers that the computing unit is responsible for (for example, 10) is divisible by the number of blocks (for example, 2), can the interleaved pipeline parallel training process be performed. Otherwise, when the number of layers that the computing unit is responsible for is not divisible by the number of blocks, it is necessary to fall back to perform the non-interleaved pipeline parallel training process, which is not conducive to improving the training efficiency. For example, when Figure 2 When the number of layers in which the computing unit is responsible is 11 and the number of blocks is 2, the interleaved pipeline parallelism is not executed, and the following is executed: Figure 1 The non-interleaved pipelined parallel training process is shown.

[0043] In other words, in the distributed framework commonly used in the industry, only equally divisible interleaved pipeline parallel training processes are supported. Assume that the total number of layers of the large model is N, the number of computing units is T, and the number of layers each computing unit is responsible for is M (that is, M=N / T). In the equally divisible interleaved pipeline parallel training process, M must be divisible by the number of blocks (that is, the result of M divided by the number of blocks is an integer with no remainder), and the number of layers responsible for each block must be the same. If the number of layers M that the computing device is responsible for cannot be divided by the number of blocks, interleaved pipeline parallel training cannot be used, and can only fall back to non-interleaved pipeline parallel training, which limits the adaptability of interleaved pipeline parallel training scenarios.

[0044] The factors that affect the training process mainly include: (1) bubble time, where finer-grained block segmentation can reduce the bubble time generated by pipeline parallelism; (2) video memory, the number of blocks will affect the video memory consumption of each computing unit. In the embodiment of the present invention, the constraint that the number of layers responsible for each computing unit must be divisible by the number of blocks is broken, and an interleaved pipeline parallel training process that supports both equal and unequal blocks is realized. In the embodiment of the present invention, according to the user's configuration, combined with the parameters of the current large model (for example, the total number of model layers and the number of computing units, etc.) and the video memory capacity on the computing unit, according to the video memory consumption of the current model and bubble time and other factors, the block size (that is, the number of layers contained in each block) in the interleaved pipeline parallel training process that meets the user's expectations can be automatically allocated.

[0045] The above disclosure details the technical defects existing in the related art, the causes of the technical defects, and the thinking and analysis process of overcoming the technical defects. In fact, the cognition of the above technical defects is not common knowledge in the field, but a novel discovery made by the inventor in his research. In addition, the cause tracing of the technical defects and the thinking and analysis process of overcoming the technical defects are also the gradual analysis results of the inventor in the actual research process, and are not common knowledge in the field.

[0046] Figure 3 4 is an exemplary flow chart of an interleaved pipeline parallel training method according to an embodiment of the present invention.

[0047] like Figure 3 As shown, the method includes: Step 101: Based on the number of computing units, the number of blocks and the total number of layers of the large model, determine the number of layers contained in each block, wherein the number of layers contained in each block is configurable. Based on the number of computing units, the number of blocks and the total number of layers of the large model, determine the number of layers contained in each block, including one of the following: based on the number of computing units, the number of blocks and the total number of layers, determine the number of layers contained in each block with bubble time as an indicator; based on the number of computing units, the number of blocks and the total number of layers, determine the number of layers contained in each block with video memory consumption as an indicator; based on the number of computing units, the number of blocks and the total number of layers, determine the number of layers contained in each block with bubble time and video memory consumption as indicators.

[0048] Here, the number of computational units and the total number of layers of the large model are known parameters. The number of blocks can be specified by the user or can be a specified default value (the default value can be changed). The number of layers contained in each block is configurable. In other words, the number of layers contained in each block can be the same, or the number of layers contained in at least two blocks can be different.

[0049] In one embodiment, before step 101, the method further includes: determining the number of layers of each computing unit based on the total number of layers and the number of computing units; determining the number of layers contained in each block based on the number of computing units, the number of blocks and the total number of layers of the large model, including: when the number of layers of each computing unit is not divisible by the number of blocks, the number of layers contained in at least two blocks is different; when the number of layers of each computing unit is divisible by the number of blocks, the number of layers contained in each block is the same.

[0050] For example: Assume that the total number of layers of the large model is N, the number of computing units is T, and the number of layers each computing unit is responsible for is M (that is, M=N / T). When the number of layers M of each computing unit is not divisible by the number of blocks, at least two blocks contain different numbers of layers; when the number of layers of each computing unit is divisible by the number of blocks, the number of layers contained in each block is the same.

[0051] Therefore, in the implementation mode of the present invention, the constraint that the number of layers responsible for each computing unit must be divisible by the number of blocks is broken, and the number of layers contained in each block is configurable, thereby supporting both evenly divided blocks and unequally divided blocks in an interleaved pipeline parallel training process.

[0052] When the number of layers M of each computing unit is not divisible by the number of blocks, at least two blocks contain different numbers of layers. In this case, the number of layers contained in each block can be specifically determined based on the bubble time or the video memory consumption.

[0053] In one embodiment, step 101 includes one of the following: Method (1): Based on the number of calculation units, the number of blocks and the total number of layers, the number of layers contained in each block is determined using the bubble time as an indicator.

[0054] In method (1), the number of layers contained in each block is determined based on the bubble time as an indicator. Generally, after the number of layers contained in the block is determined, the bubble time is minimized.

[0055] In one embodiment, method (1) specifically includes: based on the number of computing units and the number of blocks, determining multiple candidate solutions for allocating the total number of layers to each block, each candidate solution including the number of layers included in each block; determining the bubble time of each candidate solution; selecting a candidate solution with the shortest bubble time from the multiple candidate solutions; and determining the number of layers included in each block based on the candidate solution with the shortest bubble time.

[0056] In one embodiment, determining the bubble time of each candidate solution includes: determining the forward time of each block of each candidate solution to perform forward calculation on a unit micro-batch; determining the maximum forward time from multiple forward times of multiple blocks of each candidate solution; determining the reverse time of each block of each candidate solution to perform reverse calculation on a unit micro-batch; determining the maximum reverse time from multiple reverse times of multiple blocks of each candidate solution; and determining the bubble time of each candidate solution based on the maximum forward time, the maximum reverse time, and the number of computing units. Therefore, the quantification process of the bubble time can be implemented based on the maximum forward time, the maximum reverse time, and the number of computing units.

[0057] The following exemplary description discusses a typical process for quantitatively evaluating the bubble time of candidate solutions.

[0058] If the number of layers contained in each block is not limited to achieve uneven block distribution, the size of the bubble time will be determined by the block with the most layers. The total forward time of all layers that the computing unit is responsible for is equal to the sum of the forward times of all blocks, and the total reverse time of all layers is equal to the sum of the reverse times of all blocks. For interleaved pipeline scheduling with uneven block distribution, the total bubble time will be determined by the block with the longest time consumption (i.e., the block with the most layers).

[0059] Therefore, any T The total bubble time of the calculation unit ( ) and the longest forward time among all blocks ( ) and the longest reverse time among all blocks ( ) is related to the sum result of .

[0060] ;Formula (1) in: For the T The computational unit i The forward time of a micro-batch of chunks; For the T The computational unit i The reverse time of a micro-batch of chunks; is the number of calculation units; max To find the maximum value function; i The sequence number of the block.

[0061] Then, the sum of the total bubble times of all computing units in each candidate solution is calculated, that is, the bubble time of each candidate solution.

[0062] For example, suppose the total number of layers in the large model is 20, the number of computing units is 4, and each computing unit is responsible for computing 20 / 4=5 layers. Suppose the number of blocks is 2, called block 1 and block 2. Determine multiple candidate solutions for allocating the number of layers (5) that each computing unit is responsible for computing to block 1 and block 2.

[0063] For example, candidate solution 1: block 1 contains 1 layer, block 2 contains 4 layers; candidate solution 2: block 1 contains 2 layers, block 2 contains 3 layers; candidate solution 3: block 1 contains 3 layers, block 2 contains 2 layers; candidate solution 4: block 1 contains 4 layers, block 2 contains 1 layer, and so on.

[0064] Then, the bubble time of each of the five candidate solutions is calculated based on formula (1), and the candidate solution with the shortest bubble time is selected from the five candidate solutions. Assuming that the bubble time of candidate solution 2 is the shortest, it can be determined that block 1 contains 2 layers and block 2 contains 3 layers.

[0065] Method (2): Based on the number of computing units, the number of blocks, and the total number of layers, the number of layers in each block is determined using video memory consumption as an indicator.

[0066] In method (2), the number of layers contained in each block is determined based on the memory consumption as an indicator. Generally, after the number of layers contained in the block is determined, the memory consumption is minimized.

[0067] In one embodiment, method (2) specifically includes: based on the number of computing units and the number of blocks, determining multiple candidate schemes for allocating the total number of layers to each block, each candidate scheme including the number of layers included in each block; determining the video memory consumption of the computing unit that first performs calculations in each candidate scheme; selecting a candidate scheme with the smallest video memory consumption from the multiple candidate schemes; and determining the number of layers included in each block based on the candidate scheme with the smallest video memory consumption.

[0068] In one embodiment, determining the video memory consumption of the computing unit that first performs calculations in each candidate solution includes: determining the first activation value of all blocks of the computing unit that first performs calculations in each candidate solution for performing calculations on a unit micro-batch; determining the second activation value of the first block that first performs calculations in the computing unit that first performs calculations in each candidate solution for performing calculations on a unit micro-batch; and determining the video memory consumption of the computing unit that first performs calculations in each candidate solution based on the first activation value, the second activation value, and the number of computing units. Therefore, the quantification process of video memory consumption can be implemented based on the first activation value, the second activation value, and the number of computing units.

[0069] The following exemplary description discusses how to quantitatively evaluate the video memory consumption of candidate solutions.

[0070] Large model memory is usually divided into static memory and dynamic memory. Usually, static memory is the memory of the resident computing unit during the training process, including the parameters that need to be trained, the optimizer state required by the optimizer, etc. The consumption of dynamic memory is usually characterized by activation values, which are dynamically applied and released during the training process. Generally speaking, the forward activation value needs to be held until the reverse is consumed. For pipeline parallelism, in order to minimize bubbles, the reverse calculation of the first micro-batch and the forward calculation of the first micro-batch usually include multiple micro-batches of forward calculations. Therefore, in the absence of recalculation, starting from the warm-up stage, a large number of activation values ​​will need to be held until the first reverse calculation begins.

[0071] Moreover, the memory usage of activation values ​​is usually different for different computational units, with the first computational unit usually occupying more.

[0072] For staggered pipeline parallel training, due to finer-grained segmentation and the pursuit of fewer bubbles, the video memory occupancy of each computing unit is more than that of the ordinary 1F1B.

[0073] Based on formula (2), the calculation unit that first performs the calculation in each candidate solution (such as Figure 1 and Figure 2 The video memory consumption of device A).

[0074] ;Formula (2) in: The video memory consumption of the CU that first performs the calculation (that is, the first CU); a first activation value for performing computation on a unit mini-batch for all blocks of computational units that first perform computation; The first block of the computational unit that first performs the computation (i.e., the first computational unit) chunk 1 ) A second activation value for performing computation on the unit mini-batch; is the number of calculation units.

[0075] For example, suppose the total number of layers in the large model is 20, the number of computing units is 4, and each computing unit is responsible for computing 20 / 4=5 layers. Suppose the number of blocks is 2, called block 1 and block 2. Determine multiple candidate solutions for allocating the number of layers (5) that each computing unit is responsible for computing to block 1 and block 2.

[0076] For example, candidate solution 1: block 1 contains 1 layer, block 2 contains 4 layers; candidate solution 2: block 1 contains 2 layers, block 2 contains 3 layers; candidate solution 3: block 1 contains 3 layers, block 2 contains 2 layers; candidate solution 4: block 1 contains 4 layers, block 2 contains 1 layer, and so on.

[0077] Then, the memory consumption of the computing unit that first performs the calculation of the above five candidate solutions is calculated based on formula (2), and the candidate solution with the least memory consumption is selected from the above five candidate solutions. Assuming that the memory consumption of candidate solution 1 is the least, it is determined that block 1 contains 1 layer and block 2 contains 4 layers.

[0078] Method (3): Based on the number of computing units, the number of blocks, and the total number of layers, the number of layers contained in each block is determined using bubble time and video memory consumption as indicators.

[0079] In one embodiment, method (3) specifically includes: based on the number of computing units and the number of blocks, determining multiple candidate schemes for allocating the total number of layers to each block, each candidate scheme including the number of layers included in each block; determining the video memory consumption of the computing unit that first performs calculations in each candidate scheme; selecting at least one candidate scheme whose video memory consumption is lower than a first predetermined threshold from the multiple candidate schemes; determining the bubble time of at least one candidate scheme; selecting the candidate scheme with the shortest bubble time from at least one candidate scheme; and determining the number of layers included in each block based on the candidate scheme with the shortest bubble time.

[0080] It can be seen that based on the video memory consumption as the first filtering index, at least one candidate solution whose video memory consumption meets expectations is screened out, and then the bubble time is used as the selection index to select the candidate solution with the shortest bubble time from the at least one candidate solution.

[0081] For example: Assume that the total number of layers of the large model is 20, the number of computing units is 4, and the number of layers that each computing unit is responsible for computing is 20 / 4=5 layers. Assume that the number of blocks is 2, called block 1 and block 2. Determine multiple candidate solutions for allocating the number of layers (5) that each computing unit is responsible for computing to block 1 and block 2. For example, candidate solution 1: block 1 contains 1 layer, block 2 contains 4 layers; candidate solution 2: block 1 contains 2 layers, block 2 contains 3 layers; candidate solution 3: block 1 contains 3 layers, block 2 contains 2 layers; candidate solution 4: block 1 contains 4 layers, block 2 contains 1 layer, and so on. Based on formula (2), calculate the video memory consumption of the computing unit that first performs the calculation of the above five candidate solutions, and select at least one candidate solution whose video memory consumption meets the expectations from the above five candidate solutions, assuming that candidate solution 1, candidate solution 2 and candidate solution 3 are selected. Based on formula (1), the bubble time of candidate solution 1, candidate solution 2 and candidate solution 3 is calculated, and then the candidate solution with the shortest bubble time is selected from candidate solution 1, candidate solution 2 and candidate solution 3. Assuming it is candidate solution 2, it is determined that block 1 contains 2 layers and block 2 contains 3 layers.

[0082] In one embodiment, method (3) specifically includes: based on the number of computing units and the number of blocks, determining multiple candidate schemes for allocating the total number of layers to each block, each candidate scheme including the number of layers included in each block; determining the bubble time of each candidate scheme; selecting at least one candidate scheme whose bubble time is lower than a second predetermined threshold from the multiple candidate schemes; determining the video memory consumption of the computing unit that first performs the calculation in at least one candidate scheme; selecting the candidate scheme with the smallest video memory consumption from at least one candidate scheme; and determining the number of layers included in each block based on the candidate scheme with the smallest video memory consumption.

[0083] For example: Assume that the total number of layers of the large model is 20, the number of computing units is 4, and the number of layers that each computing unit is responsible for computing is 20 / 4=5 layers. Assume that the number of blocks is 2, called block 1 and block 2. Determine multiple candidate solutions for allocating the number of layers (5) that each computing unit is responsible for computing to block 1 and block 2. For example, candidate solution 1: block 1 contains 1 layer, block 2 contains 4 layers; candidate solution 2: block 1 contains 2 layers, block 2 contains 3 layers; candidate solution 3: block 1 contains 3 layers, block 2 contains 2 layers; candidate solution 4: block 1 contains 4 layers, block 2 contains 1 layer, and so on. Based on formula (1), calculate the bubble time of the above five candidate solutions, and select at least one candidate solution whose bubble time meets the expectations from the above five candidate solutions. Assume that candidate solution 2, candidate solution 3 and candidate solution 4 are selected. Then, based on formula (2), the video memory consumption of the computing unit that first performs the calculation among candidate solutions 2, candidate solutions 3 and candidate solutions 4 is calculated, and then the candidate solution with the least video memory consumption of the computing unit that first performs the calculation is selected from candidate solutions 2, candidate solutions 3 and candidate solutions 4. Assuming it is candidate solution 3, it is determined that block 1 contains 3 layers and block 2 contains 2 layers.

[0084] It can be seen that the bubble time is used as the first filtering indicator to screen out at least one candidate solution whose bubble time meets expectations, and then the video memory consumption is used as the selection indicator to select the candidate solution with the least video memory consumption from the at least one candidate solution.

[0085] In one embodiment, method (3) specifically includes: based on the number of computing units and the number of blocks, determining multiple candidate schemes for allocating the total number of layers to each block, each candidate scheme including the number of layers included in each block; determining the bubble time of each candidate scheme and the video memory consumption of the computing unit that first performs the calculation in each candidate scheme; selecting at least one candidate scheme from the multiple candidate schemes whose video memory consumption is lower than a first predetermined threshold and whose bubble time is lower than a second predetermined threshold; selecting a candidate scheme with the smallest video memory consumption or the smallest bubble time from at least one candidate scheme; and determining the number of layers included in each block based on the selected candidate scheme.

[0086] It can be seen that based on the bubble time and video memory consumption as selection indicators, the candidate solution whose bubble time and video memory consumption meet the expectations is selected.

[0087] For example: Assume that the total number of layers of the large model is 20, the number of computing units is 4, and the number of layers that each computing unit is responsible for computing is 20 / 4=5 layers. Assume that the number of blocks is 2, called block 1 and block 2. Determine multiple candidate solutions for allocating the number of layers (5) that each computing unit is responsible for computing to block 1 and block 2. For example, candidate solution 1: block 1 contains 1 layer, block 2 contains 4 layers; candidate solution 2: block 1 contains 2 layers, block 2 contains 3 layers; candidate solution 3: block 1 contains 3 layers, block 2 contains 2 layers; candidate solution 4: block 1 contains 4 layers, block 2 contains 1 layer, and so on. Based on formula (1), calculate the bubble time of the above five candidate solutions, and based on formula (2), calculate the video memory consumption of the computing unit that first performs the calculation of the above five candidate solutions, and select the candidate solutions whose bubble time and video memory consumption meet the expectations from the above five candidate solutions. Assume that candidate solutions 3 and candidate solutions 4 are selected. Then, based on the minimum bubble time, candidate solution 3 is selected from candidate solutions 3 and candidate solutions 4, and it is determined that block 1 contains 3 layers and block 2 contains 2 layers.

[0088] The above demonstration describes a typical process of determining the number of layers contained in each block based on the number of computing units, the number of blocks, and the total number of layers of the large model. Those skilled in the art will appreciate that this description is merely exemplary and is not intended to limit the scope of protection of the embodiments of the present invention.

[0089] Step 102: Based on the number of layers contained in each block, perform staggered pipeline parallel training on the large model.

[0090] Here, an interleaved pipeline parallel training is performed on the large model based on the number of layers contained in each block determined in step 101. Each computing unit in the pipeline is assigned multiple pipeline stages. Compared with the non-interleaved method, the amount of computation in each pipeline stage is less.

[0091] Figure 4 is an exemplary schematic diagram of an interleaved pipeline parallel training process according to an embodiment of the present invention.

[0092] exist Figure 4 In the example, four computing units A~D are used for description. Assuming that the number of micro-batches that can be processed in one interleaved scheduling is 8 (that is, the number of accumulations is 8), and the total number of layers of the large model is 40, then the number of layers that each computing unit is responsible for calculating is 40 / 4=10 layers. Assuming that the number of blocks is 2, called block 1 and block 2. Assuming that based on Figure 3The method shown uses bubble time or video memory consumption as an indicator to determine that block 1 contains 3 layers and block 2 contains 7 layers. It can be seen that the number of layers contained in block 1 and block 2 is not the same. The layers that computing devices A to D are responsible for are determined respectively. Among them: based on the order of computing devices and in the order of layer numbers from small to large, the first 3 layers (because block 1 contains 3 layers) are allocated in sequence in the first executed block (block 1) of each computing device, and the latter 7 layers (because block 2 contains 7 layers) are allocated in sequence in the later executed block (block 1). For example, for computing device A, block 1 is responsible for layers 1 to 3, and block 2 is responsible for layers 13 to 19; for computing device B, block 1 is responsible for layers 4 to 6, and block 2 is responsible for layers 20 to 26; for computing device C, block 1 is responsible for layers 7 to 9, and block 2 is responsible for layers 27 to 33; for computing device D, block 1 is responsible for layers 10 to 12, and block 2 is responsible for layers 34 to 40.

[0093] Taking computing unit A as an example, computing unit A should be responsible for the calculation of layers 1 to 3 and layers 13 to 19. Execute in sequence: (1) Forward operation of block 1 (including layers 1 to 3) of micro-batch numbers 1 to 4 ( Figure 4 (2) Forward operation of block 2 (including layers 13 to 19) with mini-batch numbers 1 to 4 ( Figure 4 (3) Forward operation of block 1 (including layers 1 to 3) with micro-batch numbers 5 to 6 ( Figure 4 (4) Waiting (i.e., bubble time, Figure 4 (5) Forward operation of block 1 (including layers 1 to 3) of micro-batch number 7 ( Figure 4 (6) Reverse operation of block 2 (including layers 13 to 19) of micro-batch number 1 ( Figure 4 (7) Forward operation of block 1 (including layers 1 to 3) of micro-batch number 8 ( Figure 4 The dark blue box with the number 8 is marked in the figure); (8): Waiting (i.e., bubble time, Figure 4 (9) Reverse operation of block 2 (including layers 13 to 19) of micro-batch number 2 ( Figure 4 (10) Forward operation of block 2 (including layers 13 to 19) of micro-batch number 5 ( Figure 4 (11) Reverse operation of block 2 (including layers 13 to 19) of micro-batch number 3 ( Figure 4(12) Forward operation of block 2 (including layers 13 to 19) of micro-batch number 6 ( Figure 4 (13) Reverse operation of block 2 (including layers 13 to 19) of micro-batch number 4 ( Figure 4 (14) Forward operation of block 2 (including layers 13 to 19) of micro-batch number 7 ( Figure 4 (15) Reverse operation of block 1 (including layers 1 to 3) of micro-batch number 1 ( Figure 4 dark green box with the number 1 in the figure); ... and so on.

[0094] Taking calculation unit B as an example, calculation unit B should be responsible for the calculation of layers 4 to 6 and layers 20 to 26. Execute in sequence: (1) Wait (i.e., bubble time, Figure 4 (2) Forward operation of block 1 (including layers 4 to 6) of micro-batch numbers 1 to 4 ( Figure 4 (3) Waiting (i.e., bubble time, Figure 4 (4) Forward operation of block 2 (including layers 20 to 26) of micro-batch numbers 1 to 4 ( Figure 4 (5) Waiting (i.e., bubble time, Figure 4 (6) Forward operation of block 1 (including layers 4 to 6) of micro-batch number 5 ( Figure 4 The light blue box with the number 5 in it); ... and so on.

[0095] Similarly, computational units C~D are assigned two unequally divided blocks and are responsible for the computation of their own layers, thus allowing interleaved execution of forward and reverse between different stages.

[0096] It can be seen that the unequally divided blocks realize interleaved pipeline parallel training without falling back to non-interleaved pipeline parallel training, thereby improving the training efficiency.

[0097] Figure 5 FIG. 1 is an exemplary structural diagram of an interleaved pipeline parallel training device according to an embodiment of the present invention. Figure 5As shown, the staggered pipeline parallel training device 200 includes: a determination module 201, which is used to determine the number of layers contained in each block based on the number of computing units, the number of blocks and the total number of layers of the large model, wherein the number of layers contained in each block is configurable; a training module 202, which is used to perform staggered pipeline parallel training on the large model based on the number of layers contained in each block; the determination module 201 is used to perform at least one of the following: based on the number of computing units, the number of blocks and the total number of layers, the number of layers contained in each block is determined with the bubble time as an indicator; based on the number of computing units, the number of blocks and the total number of layers, the number of layers contained in each block is determined with the video memory consumption as an indicator; based on the number of computing units, the number of blocks and the total number of layers, the number of layers contained in each block is determined with the bubble time and video memory consumption as indicators. In one embodiment, the determination module 201 is used to determine multiple candidate schemes for allocating the total number of layers to each block based on the number of computing units and the number of blocks, each candidate scheme including the number of layers contained in each block; determine the bubble time of each candidate scheme; select the candidate scheme with the shortest bubble time from the multiple candidate schemes; and determine the number of layers contained in each block based on the candidate scheme with the shortest bubble time.

[0098] In one embodiment, the determination module 201 is used to determine multiple candidate schemes for allocating the total number of layers to each block based on the number of computing units and the number of blocks, each candidate scheme including the number of layers contained in each block; determine the video memory consumption of the computing unit that first performs calculations in each candidate scheme; select the candidate scheme with the minimum video memory consumption from the multiple candidate schemes; and determine the number of layers contained in each block based on the candidate scheme with the minimum video memory consumption.

[0099] In one embodiment, the determination module 201 is configured to perform one of the following: (1) Based on the number of computing units and the number of blocks, determine multiple candidate solutions for allocating the total number of layers to each block, each candidate solution including the number of layers included in each block; determine the video memory consumption of the computing unit that first performs calculations in each candidate solution; select at least one candidate solution whose video memory consumption is lower than a first predetermined threshold from the multiple candidate solutions; determine the bubble time of at least one candidate solution; select a candidate solution with the shortest bubble time from the at least one candidate solution; and determine the number of layers included in each block based on the candidate solution with the shortest bubble time; (2) Based on the number of computing units and the number of blocks, determine multiple candidate solutions for allocating the total number of layers to each block, each candidate solution including the number of layers included in each block; determine the bubble time of each candidate solution; select at least one candidate solution whose bubble time is lower than a second predetermined threshold from the multiple candidate solutions; determine the video memory consumption of the computing unit that first performs the calculation in at least one candidate solution; select a candidate solution with the minimum video memory consumption from the at least one candidate solution; and determine the number of layers included in each block based on the candidate solution with the minimum video memory consumption; (3) Based on the number of computing units and the number of blocks, determine multiple candidate solutions for allocating the total number of layers to each block, each candidate solution including the number of layers included in each block; determine the bubble time of each candidate solution and the video memory consumption of the computing unit that first performs the calculation in each candidate solution; select at least one candidate solution whose video memory consumption is lower than a first predetermined threshold and whose bubble time is lower than a second predetermined threshold from the multiple candidate solutions; select a candidate solution with the smallest video memory consumption or a candidate solution with the smallest bubble time from the at least one candidate solution; and determine the number of layers included in each block based on the selected candidate solution.

[0100] In summary, in an embodiment of the present invention, the number of layers contained in each block is determined based on the number of computing units, the number of blocks and the total number of layers of the large model, wherein the number of layers contained in each block is configurable; based on the number of layers contained in each block, staggered pipeline parallel training is performed on the large model. It can be seen that the constraint that the number of layers responsible for each computing unit must be divisible by the number of blocks is broken, and a non-staggered pipeline parallel training process that supports both equal and unequal divisions is realized, which improves training efficiency and expands the applicable scenarios of staggered pipeline parallel training. In addition, a good compromise can be achieved between bubble time and video memory consumption. An embodiment of the present invention also proposes an electronic device with a processor-memory architecture. Figure 6 is a structural diagram of an electronic device according to an embodiment of the present invention. Figure 6 As shown, the electronic device includes a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the computer program is executed by the processor 301, any of the above interleaved pipeline parallel training methods is implemented. Among them, the memory 302 can be specifically implemented as a variety of storage media such as an electrically erasable programmable read-only memory (EEPROM), a flash memory (Flash memory), and a programmable program read-only memory (PROM). The processor 301 can be implemented to include one or more central processing units or one or more field programmable gate arrays, wherein the field programmable gate array integrates one or more central processing unit cores. Specifically, the central processing unit or the central processing unit core can be implemented as a CPU, a GPU, a GPGPU, an MCU or a DSP, and so on.

[0101] It should be noted that not all steps and modules in the above processes and structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The division of each module is only for the convenience of describing the functional division adopted. In actual implementation, a module can be implemented by multiple modules, and the functions of multiple modules can also be implemented by the same module. These modules can be located in the same device or in different devices.

[0102] The hardware modules in each embodiment may be implemented mechanically or electronically. For example, a hardware module may include a specially designed permanent circuit or logic device (such as a dedicated processor, such as an FPGA or ASIC) to perform a specific operation. For example, specific operations can be performed in various types of chips (for example, artificial intelligence chips). The hardware module may also include a programmable logic device or circuit (such as a general-purpose processor or other programmable processor) temporarily configured by software to perform specific operations. As for whether to implement the hardware module mechanically, or using a dedicated permanent circuit, or a temporarily configured circuit (such as configured by software), it can be decided based on cost and time considerations.

[0103] The present invention also provides a machine-readable storage medium, storing instructions for a machine to perform a method as described in the present application. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code for implementing the functions of any of the embodiments described in the above embodiments is stored, and the computer (or CPU or MPU) of the system or device reads out and executes the program code stored in the storage medium. In addition, the operating system operated on the computer can also be made to complete part or all of the actual operations by instructions based on the program code. The program code read out from the storage medium can also be written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion unit connected to the computer, and then the CPU installed on the expansion board or the expansion unit is made to perform part and all of the actual operations based on the instructions of the program code, thereby realizing the functions of any of the embodiments described in the above embodiments. The storage medium implementation for providing the program code includes a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card and a ROM. Alternatively, the program code can be downloaded from a server computer or a cloud by a communication network.

[0104] In this article, "schematic" means "serving as an example, instance or explanation", and any diagram or implementation method described as "schematic" in this article should not be interpreted as a more preferred or more advantageous technical solution. In order to make the drawings concise, only the parts related to the present invention are schematically shown in each figure, and do not represent the actual structure of the product. In addition, in order to make the drawings concise and easy to understand, in some figures, only one of the parts with the same structure or function is schematically drawn, or only one of them is marked. In this article, "one" does not mean that the number of the relevant parts of the present invention is limited to "only one", and "one" does not mean that the number of the relevant parts of the present invention is "more than one". In this article, "upper", "lower", "front", "back", "left", "right", "inside", "outside", etc. are only used to indicate the relative position relationship between the relevant parts, rather than to limit the absolute position of these relevant parts.

[0105] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A staggered pipeline parallel training method, characterized in that: include: Based on the number of computing units, the number of blocks and the total number of layers of the large model, the number of layers contained in each block is determined, wherein the number of layers contained in each block is configurable; the method of determining the number of layers contained in each block based on the number of computing units, the number of blocks and the total number of layers of the large model includes one of the following: based on the number of computing units, the number of blocks and the total number of layers, the number of layers contained in each block is determined with bubble time as an indicator; based on the number of computing units, the number of blocks and the total number of layers, the number of layers contained in each block is determined with video memory consumption as an indicator; Based on the number of computing units, the number of blocks and the total number of layers, the number of layers included in each block is determined with bubble time and video memory consumption as indicators; Based on the number of layers contained in each block, staggered pipeline parallel training is performed on the large model.

2. The method according to claim 1, characterized in that The determining the number of layers contained in each block based on the number of computing units, the number of blocks and the total number of layers and taking the bubble time as an indicator comprises: Based on the number of computing units and the number of blocks, determining a plurality of candidate solutions for allocating the total number of layers to each block, each candidate solution including the number of layers included in each block; Determine the bubble time for each candidate solution; Selecting a candidate solution with the shortest bubble time from the plurality of candidate solutions; The number of layers included in each block is determined based on the candidate solution with the shortest bubble time.

3. The method according to claim 2, characterized in that Determining the bubble time of each candidate solution includes: Determine the forward time for each candidate solution to perform forward computation on a unit micro-batch for each block; Determine a maximum forward time from the multiple forward times of the multiple blocks of each candidate solution; Determine the reverse time for each block of each candidate solution to perform the reverse calculation on the unit micro-batch; Determine a maximum reverse time from multiple reverse times of multiple blocks of each candidate solution; The bubble time of each candidate solution is determined based on the maximum forward time, the maximum reverse time and the number of computing units.

4. The method according to claim 1, characterized in that: The determining the number of layers included in each block based on the number of computing units, the number of blocks and the total number of layers and taking video memory consumption as an indicator includes: Based on the number of computing units and the number of blocks, determining a plurality of candidate solutions for allocating the total number of layers to each block, each candidate solution including the number of layers included in each block; Determine the video memory consumption of the computing unit that first performs the calculation in each candidate solution; Selecting the candidate solution with the smallest video memory consumption from the multiple candidate solutions; Based on the candidate solution with the minimum video memory consumption, the number of layers included in each block is determined.

5. The method according to claim 4, characterized in that Determining the video memory consumption of the computing unit that first performs the calculation in each candidate solution includes: Determine a first activation value for all blocks of the computing unit that first performs the calculation on the unit micro-batch in each candidate solution; Determine a second activation value of a first block of a computing unit that first performs calculations in each candidate solution for performing calculations on a unit micro-batch; Based on the first activation value, the second activation value, and the number of computing units, a video memory consumption of a computing unit that first performs calculations in each candidate solution is determined.

6. The method according to claim 1, characterized in that The determining the number of layers included in each block based on the number of computing units, the number of blocks and the total number of layers and taking bubble time and video memory consumption as indicators includes: Based on the number of computing units and the number of blocks, determining a plurality of candidate solutions for allocating the total number of layers to each block, each candidate solution including the number of layers included in each block; Determine the video memory consumption of the computing unit that first performs the calculation in each candidate solution; Selecting at least one candidate solution whose video memory consumption is lower than a first predetermined threshold from the multiple candidate solutions; determining a bubble time for the at least one candidate solution; Selecting a candidate solution with the shortest bubble time from the at least one candidate solution; The number of layers included in each block is determined based on the candidate solution with the shortest bubble time.

7. The method according to claim 1, characterized in that The determining the number of layers included in each block based on the number of computing units, the number of blocks and the total number of layers and taking bubble time and video memory consumption as indicators includes: Based on the number of computing units and the number of blocks, determining a plurality of candidate solutions for allocating the total number of layers to each block, each candidate solution including the number of layers included in each block; Determine the bubble time for each candidate solution; Selecting at least one candidate solution having a bubble time lower than a second predetermined threshold from the plurality of candidate solutions; Determine the video memory consumption of the computing unit that first performs the calculation in the at least one candidate solution; Selecting a candidate solution with the smallest video memory consumption from the at least one candidate solution; Based on the candidate solution with the minimum video memory consumption, the number of layers included in each block is determined.

8. The method according to claim 1, characterized in that The determining the number of layers included in each block based on the number of computing units, the number of blocks and the total number of layers and taking bubble time and video memory consumption as indicators includes: Based on the number of computing units and the number of blocks, determining a plurality of candidate solutions for allocating the total number of layers to each block, each candidate solution including the number of layers included in each block; Determine the bubble time of each candidate solution and the video memory consumption of the computing unit that first performs the calculation in each candidate solution; Selecting at least one candidate solution from the plurality of candidate solutions, the video memory consumption of which is lower than a first predetermined threshold and the bubble time of which is lower than a second predetermined threshold; Selecting a candidate solution with the smallest video memory consumption or a candidate solution with the smallest bubble time from the at least one candidate solution; Based on the selected candidate solution, the number of layers included in each block is determined.

9. The method according to any one of claims 1 to 8, characterized in that Before determining the number of layers contained in each block based on the number of computing units, the number of blocks, and the total number of layers of the large model, the method further includes: Determining the number of layers of each computing unit based on the total number of layers and the number of computing units; The method of determining the number of layers contained in each block based on the number of computing units, the number of blocks, and the total number of layers of the large model includes: When the number of layers of each computing unit is not divisible by the number of blocks, the number of layers included in at least two blocks is different; When the number of layers of each computing unit is divisible by the number of blocks, the number of layers contained in each block is the same.

10. An interleaved pipeline parallel training device, applied to the interleaved pipeline parallel training method according to any one of claims 1 to 9, characterized in that: include: A determination module, used for determining the number of layers contained in each block based on the number of computing units, the number of blocks and the total number of layers of the large model, wherein the number of layers contained in each block is configurable; A training module for performing staggered pipeline parallel training on the large model based on the number of layers contained in each block The determining module is used to perform one of the following: Based on the number of computing units, the number of blocks and the total number of layers, the number of layers included in each block is determined with bubble time as an indicator; Based on the number of computing units, the number of blocks and the total number of layers, the number of layers included in each block is determined with video memory consumption as an indicator; Based on the number of computing units, the number of blocks and the total number of layers, the number of layers included in each block is determined with bubble time and video memory consumption as indicators.

11. The device according to claim 10, characterized in that The determination module is used to determine multiple candidate solutions for allocating the total number of layers to each block based on the number of computing units and the number of blocks, each candidate solution including the number of layers included in each block; and determine the bubble time of each candidate solution; A candidate solution with the shortest bubble time is selected from the multiple candidate solutions; and the number of layers included in each block is determined based on the candidate solution with the shortest bubble time.

12. The device according to claim 10, characterized in that The determination module is used to determine multiple candidate solutions for allocating the total number of layers to each block based on the number of computing units and the number of blocks, each candidate solution including the number of layers included in each block; determine the video memory consumption of the computing unit that first performs calculation in each candidate solution; and select the candidate solution with the minimum video memory consumption from the multiple candidate solutions; Based on the candidate solution with the minimum video memory consumption, the number of layers included in each block is determined.

13. The device according to claim 10, characterized in that The determining module is used to perform one of the following: Based on the number of computing units and the number of blocks, a plurality of candidate solutions for allocating the total number of layers to the respective blocks are determined, each candidate solution including the number of layers included in each block; a video memory consumption of a computing unit that first performs calculations in each candidate solution is determined; at least one candidate solution whose video memory consumption is lower than a first predetermined threshold is selected from the plurality of candidate solutions; and a bubble time of the at least one candidate solution is determined; Selecting a candidate solution with the shortest bubble time from the at least one candidate solution; Determine the number of layers included in each block based on the candidate solution with the shortest bubble time; Based on the number of computing units and the number of blocks, determine multiple candidate solutions for allocating the total number of layers to each block, each candidate solution including the number of layers included in each block; determine the bubble time of each candidate solution; Select at least one candidate solution whose bubble time is lower than a second predetermined threshold from the plurality of candidate solutions; determine the video memory consumption of the computing unit that first performs the calculation in the at least one candidate solution; Selecting a candidate solution with the smallest video memory consumption from the at least one candidate solution; Based on the candidate solution with the minimum video memory consumption, determine the number of layers included in each block; Based on the number of computing units and the number of blocks, determine multiple candidate schemes for allocating the total number of layers to each block, each candidate scheme including the number of layers included in each block; determine the bubble time of each candidate scheme and the video memory consumption of the computing unit that first performs calculations in each candidate scheme; select at least one candidate scheme whose video memory consumption is lower than a first predetermined threshold and whose bubble time is lower than a second predetermined threshold from the multiple candidate schemes; select a candidate scheme with the smallest video memory consumption or a candidate scheme with the smallest bubble time from the at least one candidate scheme; based on the selected candidate scheme, determine the number of layers included in each block.

14. An electronic device, characterized in that: include: Memory; processor; The memory stores an application program executable by the processor, which is used to enable the processor to execute the interleaved pipeline parallel training method as described in any one of claims 1-9.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the interleaved pipeline parallel training method according to any one of claims 1 to 9.

16. A program product comprising a computer program, characterized in that When the computer program is executed by a processor, the interleaved pipeline parallel training method according to any one of claims 1 to 9 is implemented.

Images