Assembly line parallel division and memory optimization method for large-scale model training

Abstract

The invention provides an assembly line parallel division and memory optimization method for large-scale model training. The method comprises the following steps: S1, identifying and dividing model operators; s2, calculating operator memory occupation and numbering; s3, constructing a target function, and minimizing a memory difference value through dynamic programming; s4, starting from a single operator, gradually constructing and searching an optimal division strategy; and S5, analyzing and optimizing memory occupation, and balancing the memory of the equipment in each stage. According to the method, automatic division of the model is realized by using a dynamic programming algorithm, and the problem of unbalanced memory occupation in each stage in pipeline parallelism is solved. And the search space of a dynamic programming algorithm is reduced by utilizing the isomorphism between model layers, so that the time required for searching an optimal solution is reduced. And the minimum unit of model division is reduced to operators, so that the model division fineness is improved, and the algorithm can search a better division strategy.

Description

Technical Field

[0001] The present invention relates to machine learning and distributed computing technologies, and in particular to a pipeline parallel partitioning and memory optimization method for large-scale model training. Background Art

[0002] Pipeline parallelism is a key technology for training large-scale models in distributed machine learning. Its main idea is to place different layers of the model on different computing devices to reduce the memory consumption of a single computing device and achieve ultra-large-scale model training. This method is particularly important when dealing with large deep learning models containing billions or even trillions of parameters, such as the current mainstream Transformer models BERT Large and GPT.

[0003] Pipeline parallelism involves dividing the computational graph into sequential subgraphs and assigning the subgraphs to different stages. In the forward pass, each sample will flow from the first stage to the last stage. In the backward pass, the gradient of each sample is back-propagated from the last stage to the first stage. The computation of a single sample exhibits sequential dependencies across stages, which means that a sample cannot be processed by two stages at the same time.

[0004] The main challenges facing pipeline parallelism include improving training speed, optimizing memory allocation, and scalability of the structure.

[0005] There have been some attempts and some progress. Among them, the GPipe pipeline parallel framework proposed by Google in 2018 is one of the most famous works. This work reduces pipeline bubbles caused by the sequential dependency of calculations by decomposing batch data into smaller micro-batches, thereby improving training efficiency. However, the activation values ​​generated in the forward pass need to be retained during the calculation process until their corresponding backward pass is completed, which results in a large amount of memory consumption. Micro-batches require more activation values ​​to be saved in the earlier stages, which means that these stages need to occupy more memory. This memory imbalance increases with the increase in the number of micro-batches / stages.

[0006] In order to solve the imbalance of memory usage, the PipeDream framework proposed by Microsoft and Stanford University in 2019 uses 1F1B (one-forward-one-backward), which releases the saved activation values ​​in advance by staggered forward and backward propagation. However, this static method still brings a certain degree of memory imbalance and cannot solve the problem that the imbalance increases with the number of stages.

[0007] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0008] The main purpose of the present invention is to solve the problems existing in the above-mentioned background technology and to provide a pipeline parallel partitioning and memory optimization method for large-scale model training.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] In a first aspect of the present invention, a pipeline parallel partitioning and memory optimization method for large-scale model training comprises the following steps:

[0011] S1. Identify each operator in the model and divide the model into multiple parts according to the fineness of the operator;

[0012] S2. Calculate the memory occupied by each operator in each round of training, number the operators according to the forward propagation order of the model, build a memory usage data model, and provide input for the dynamic programming algorithm;

[0013] S3. Construct an objective function for model partitioning to minimize the maximum difference in memory usage on devices at different stages through the objective function, and establish a recursive calling relationship of a dynamic programming algorithm to facilitate the search for a model partitioning strategy;

[0014] S4. Using dynamic programming algorithm, starting from the most basic partitioning unit, i.e., a single operator, gradually builds a more complex model partitioning strategy, and searches and determines the optimal model partitioning scheme through recursive calls and difference minimization strategies;

[0015] S5. According to the optimal model partitioning scheme, analyze the memory usage of each stage, identify the problem of unbalanced memory usage, and adjust the model partitioning strategy to optimize the memory usage so that the memory usage of the devices in each stage is as balanced as possible.

[0016] In a second aspect of the present invention, a computer program product includes a computer program, which, when executed by a processor, implements the pipeline parallel partitioning and memory optimization method for large-scale model training.

[0017] The present invention has the following beneficial effects:

[0018] The present invention proposes a pipeline parallel partitioning and memory optimization method for large-scale model training, which innovatively solves the challenges of uneven memory usage and improved training efficiency faced by pipeline parallelism in large-scale model training. The present invention realizes automatic partitioning of the model, optimizes memory usage, improves training speed, and enhances the scalability of model training through a dynamic programming algorithm. The algorithm can utilize the isomorphism between model substructures to reduce the computing time required to search for the optimal solution, while accurately dividing the smallest unit of the model to the operator level, thereby improving the precision of the model partitioning. The method of the present invention not only reduces the complexity and cost of manual model partitioning, but also significantly reduces the overall training time by reducing the memory usage of the front stage and the recalculation ratio of the back stage. In addition, the algorithm of the present invention can be effectively combined with other distributed parallel technologies such as data parallelism and tensor parallelism to further improve the efficiency of distributed training, making the present invention particularly important and effective when processing large deep learning models containing billions or even trillions of parameters.

[0019] Compared with the traditional method, the main advantages of the present invention are:

[0020] (1) The present invention can provide an automatic partitioning algorithm for model partitioning in large model parallel training, reducing the cost required for manual partitioning and greatly lowering the entry threshold for model partitioning.

[0021] (2) Use a dynamic programming algorithm to automatically partition the model and improve the imbalance in memory usage on devices at different stages under the condition of limited training time.

[0022] (3) By reducing the number of layers saved in the early stages, the memory overhead of these stages in model training can be reduced, and the batch size of hyperparameters in training can be increased, thereby reducing the training time of the model on the full dataset.

[0023] (4) By reducing the proportion of recalculation in the later stages of back propagation, the additional computational and time overhead required for recalculation in these stages can be reduced, thereby reducing training time.

[0024] (5) The algorithm is scalable and can be effectively combined with other technologies to further improve the efficiency of distributed training.

[0025] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Schematic diagram of pipeline parallel model division according to an embodiment of the present invention.

[0027] Figure 2This is a simplified flowchart of the pipeline parallel partitioning and memory optimization method for distributed parallel training of a model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0029] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0030] See also Figure 1 and Figure 2 The embodiment of the present invention provides a pipeline parallel partitioning and memory optimization method for large-scale model training, comprising the following steps:

[0031] S1. Identify the operators in the model, including basic computing units such as matrix multiplication and addition operations, and divide the model into multiple parts according to the fineness of the operators to facilitate subsequent memory usage calculation and model partitioning strategy design;

[0032] S2. Calculate the memory occupied by each operator in each round of training, including parameters, intermediate data, and activation values, and number the operators according to the forward propagation order of the model, build a memory usage data model, and provide input for the dynamic programming algorithm;

[0033] S3. Construct an objective function for model partitioning to minimize the maximum difference in memory usage on devices at different stages through the objective function, and establish a recursive calling relationship of a dynamic programming algorithm to facilitate the search for a model partitioning strategy;

[0034] S4. Using dynamic programming algorithm, starting from the most basic partitioning unit, i.e., a single operator, gradually builds a more complex model partitioning strategy, and searches and determines the optimal model partitioning scheme through recursive calls and difference minimization strategies;

[0035] S5. According to the optimal model partitioning scheme, analyze the memory usage of each stage, identify the problem of unbalanced memory usage, and adjust the model partitioning strategy to optimize the memory usage so that the memory usage of the devices in each stage is as balanced as possible.

[0036] The present invention uses a dynamic programming algorithm to achieve automatic model partitioning. It solves the problem of uneven memory usage at each stage in pipeline parallelism. It reduces the search space of the dynamic programming algorithm by utilizing the isomorphism between model layers, thereby reducing the time required to search for the optimal solution. It reduces the minimum unit of model partitioning to an operator, improves the precision of model partitioning, and enables the algorithm to search for a better partitioning strategy.

[0037] In some embodiments, in step S3, the number of stages p used in the training process is determined, where each stage corresponds to a computing stage in the pipeline parallelism; the number of groups of parameters, memory, and activation values ​​that the device at the sth stage needs to save is analyzed, which is ps, to identify the memory usage requirements of each stage; in step S4, based on the 1F1B (one-forward-one-backward) pipeline parallel scheduling strategy, the model is sequentially divided using a dynamic programming algorithm so that the device at each stage can effectively process the model part it is responsible for under the 1F1B pipeline parallel scheduling strategy; under the guidance of this strategy, the model partitioning strategy is optimized so that the stage with a smaller sequence number, i.e., s, can efficiently process more forward computing tasks while reducing the memory usage of these stages; in step S5, under the 1F1B pipeline parallel scheduling strategy, the memory usage of each stage is dynamically adjusted to adapt to the changes in memory requirements of different stages during forward and backward propagation; especially for stages with smaller sequence numbers, the memory overhead of these stages in model training is reduced by optimizing the model partitioning strategy, so as to achieve a balance in the memory usage of devices at each stage.

[0038] In some embodiments, step S2 specifically includes: dividing the model into basic computing units, such as operators such as matrix multiplication and addition operations; numbering the operators according to the forward propagation order of the model; calculating the memory occupied by each operator during the training process, and constructing a memory occupancy data model. Step S3 specifically includes: determining the goal of model partitioning, that is, minimizing the maximum difference in memory occupancy on devices at different stages; designing a dynamic programming algorithm to realize the search of model partitioning strategy. Step S4 specifically includes: using a dynamic programming algorithm, starting from a single operator, and gradually building a model partitioning strategy; searching and determining the optimal model partitioning scheme through recursive calls and difference minimization strategies. Step S5 specifically includes: using the output of the dynamic programming algorithm to determine the device to which each model layer should be assigned, and the memory occupancy that should be on each device; analyzing the memory occupancy of devices at each stage under the optimal model partitioning scheme, and identifying the problem of unbalanced memory occupancy; adjusting the model partitioning strategy for the identified unbalanced memory occupancy problem, and optimizing the memory occupancy of devices at each stage to achieve a balance in overall memory occupancy.

[0039] In some embodiments, in step S3, the optimization objective function F is defined as minimizing the maximum difference D(G) in memory usage on devices at different stages, where G represents the computational graph partitioning of a model; a recursive relationship is set, and the model partitioning is gradually optimized by comparing the memory usage at different stages until a global optimal solution is reached; through recursive calls, starting from the most basic partitioning unit, more complex model partitioning strategies are gradually constructed until the entire model is covered.

[0040] Furthermore, the method also includes: identifying substructures in the model, such as attention layers and feedforward layers, and evaluating their isomorphism; using the isomorphism of the substructures to simplify the search space of the dynamic programming algorithm, reduce the computation time, and improve the efficiency of the algorithm. The model may include a large Transformer model with attention layer and feedforward layer substructures.

[0041] In some embodiments, the method further includes: analyzing and adjusting the batch size according to memory usage and model partitioning strategy to improve training efficiency; reducing the recalculation ratio in the later stages in back propagation by optimizing the recalculation strategy, thereby reducing additional calculation and time overhead.

[0042] In some embodiments, the method further includes: verifying the effectiveness of the algorithm in an actual pipeline parallel environment so that the model partitioning strategy can actually improve training efficiency and memory utilization; and tuning the algorithm based on the verification results to adapt to different model sizes and training requirements.

[0043] The method is scalable and can be combined with other distributed parallel techniques such as data parallelism (DP) and tensor parallelism (TP); and allows differentiated recomputation strategies to be set between different layers of the model according to specific performance and resource requirements.

[0044] The specific embodiments of the present invention are further described below.

[0045] A pipeline parallel partitioning and memory optimization method for large-scale model training, in which an algorithm for automatic model partitioning in pipeline parallelism is designed, which contributes to solving the problem of unbalanced device memory usage at different stages in pipeline parallelism. The algorithm of this embodiment is preferably based on the 1F1B pipeline parallel scheduling strategy, on which a dynamic programming algorithm is used to search for the model partitioning strategy and find an approximate optimal solution to complete the sequential segmentation of the model.

[0046] Figure 1The pipeline parallel model partitioning of the embodiment of the present invention is exemplified. Assuming that there are p stages in training, in the 1F1B strategy, the device in the s stage needs to save (ps) sets of parameters, memory and activation values. The above figure shows a schematic diagram with 4 stages. The model is sequentially partitioned according to a specific strategy and deployed on stage0, stage 1, stage 2, and stage3 respectively. Figure 1 It can be seen that the stage with a smaller sequence number needs to be forwarded more times continuously, which shows that the smaller s is, the more values ​​need to be saved and the more memory is occupied.

[0047] In the designed algorithm, the model is first divided into operators with fineness, such as matrix multiplication and addition operations, and numbered [i] according to the calculation order in the network forward transmission, and the array M[i] is calculated. M[i] represents the memory occupied by operator i in each round of training. The purpose is to balance the memory usage of devices at each stage as much as possible, so the optimization goal is: in Represents the computational graph partitioning of a model, Indicates in the division The maximum difference in memory usage on the device at different stages is obviously In order to solve the minimum problem, the dynamic programming algorithm is used as follows: represents the difference corresponding to the optimal partitioning strategy when the first i operators of the model are deployed in j stages, then:

[0048]

[0049] in Indicates the maximum difference between the memory usage of different devices and M under the condition of F(i,j).

[0050] The recursive call in the knapsack problem of dynamic programming is that F(i,1) is known, and the other terms are obtainable. Assuming there are p stages and n operators, the target value to be obtained is F(n,p). The process of obtaining the above formula will continue to recurse downward until F(i,1), iin[1,n]. By recursively calling the above formula, the optimal strategy for deploying the entire model on P stages can be calculated. The algorithm outputs the device to which each model layer should be assigned, as well as the memory usage on each device.

[0051] Taking into account the large number of attention layer + feedforward layer substructures that exist in the current Transformer large model, the present invention uses the isomorphism between model substructures to reduce the complexity of the dynamic programming algorithm, thereby reducing the computing time required to search for the optimal solution.

[0052] The process of the designed algorithm is as follows Figure 2 The designed algorithm is scalable and can be combined with other distributed parallel technologies such as data parallelism (DP) and tensor parallelism (TP). Different recalculation strategies can also be set between layers as needed.

[0053] In summary, the present invention proposes a pipeline parallel partitioning and memory optimization method for large-scale model training, which uses a dynamic programming algorithm to realize automatic partitioning of the model and solves the problem of uneven memory usage at each stage in pipeline parallelism. The isomorphism between model layers is used to reduce the search space of the dynamic programming algorithm, thereby reducing the time required to search for the optimal solution. The minimum unit of model partitioning is reduced to an operator, which improves the precision of model partitioning and enables the algorithm to search for a better partitioning strategy.

[0054] Compared with the traditional method, the main advantages of the present invention are:

[0055] (1) The present invention can provide an automatic partitioning algorithm for model partitioning in large model parallel training, reducing the cost required for manual partitioning and greatly lowering the entry threshold for model partitioning.

[0056] (2) Use a dynamic programming algorithm to automatically partition the model and improve the imbalance in memory usage on devices at different stages under the condition of limited training time.

[0057] (3) By reducing the number of layers saved in the early stages, the memory overhead of these stages in model training can be reduced, and the batch size hyperparameter in training can be increased, thereby reducing the training time of the model on the full dataset.

[0058] (4) By reducing the proportion of recalculation in the later stages of back propagation, the additional computational and time overhead required for recalculation in these stages can be reduced, thereby reducing training time.

[0059] (5) The algorithm is scalable and can be effectively combined with other technologies to further improve the efficiency of distributed training.

[0060] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0061] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0062] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0063] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0064] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0065] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0066] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0067] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0068] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0069] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0070] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0071] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0072] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A pipeline parallel partitioning and memory optimization method for large-scale model training, characterized in that: The following steps are involved: S1. Identify each operator in the model and divide the model into multiple parts according to the fineness of the operator; S2. Calculate the memory occupied by each operator in each round of training, number the operators according to the forward propagation order of the model, build a memory usage data model, and provide input for the dynamic programming algorithm; S3. Construct an objective function for model partitioning to minimize the maximum difference in memory usage on devices at different stages through the objective function, and establish a recursive calling relationship of a dynamic programming algorithm to facilitate the search for a model partitioning strategy; S4. Using dynamic programming algorithm, starting from the most basic partitioning unit, i.e., a single operator, gradually builds a more complex model partitioning strategy, and searches and determines the optimal model partitioning scheme through recursive calls and difference minimization strategies; S5. According to the optimal model partitioning scheme, analyze the memory usage of each stage, identify the problem of unbalanced memory usage, and adjust the model partitioning strategy to optimize the memory usage so that the memory usage of the devices in each stage is as balanced as possible.

2. The pipeline parallel partitioning and memory optimization method for large-scale model training according to claim 1, characterized in that: In step S3, the number of stages p used in the training process is determined, where each stage corresponds to a computational stage in the pipeline parallelism; the number of groups of parameters, memory, and activation values ​​that need to be saved by the device at the sth stage is analyzed, which is ps, to identify the memory usage requirements of each stage; In step S4, based on the 1F1B pipeline parallel scheduling strategy, the model is sequentially segmented using a dynamic programming algorithm so that the device at each stage can effectively process the model part it is responsible for under the 1F1B pipeline parallel scheduling strategy; under the guidance of this strategy, the model segmentation strategy is optimized so that the stage with a smaller sequence number, i.e., s, can efficiently process more forward computing tasks, while reducing the memory usage of these stages; In step S5, under the 1F1B pipeline parallel scheduling strategy, the memory usage of each stage is dynamically adjusted to adapt to the changes in memory requirements of different stages during forward and backward propagation. In particular, for stages with smaller serial numbers, the memory overhead of these stages in model training is reduced by optimizing the model partitioning strategy, so as to achieve balanced memory usage of devices in each stage.

3. The pipeline parallel partitioning and memory optimization method for large-scale model training according to claim 1 or 2, characterized in that: Step S2 specifically includes: Divide the model into basic computing units, such as operators such as matrix multiplication and addition; Number the operators according to the forward propagation order of the model; Calculate the memory occupied by each operator during training and build a memory usage data model; Step S3 specifically includes: Determine the goal of model partitioning, that is, to minimize the maximum difference in memory usage on devices at different stages; Design a dynamic programming algorithm to search for model partitioning strategies; Step S4 specifically includes: Using dynamic programming algorithm, starting from a single operator, gradually build the model partitioning strategy; Through recursive calling and difference minimization strategy, search and determine the optimal model partitioning solution; Step S5 specifically includes: Use the output of the dynamic programming algorithm to determine the device to which each model layer should be assigned and the memory usage that each device should have; Analyze the memory usage of devices at each stage under the optimal model partitioning scheme and identify the problem of unbalanced memory usage; In response to the identified memory usage imbalance problem, the model partitioning strategy was adjusted to optimize the memory usage of devices at each stage to achieve a balance in the overall memory usage.

4. The pipeline parallel partitioning and memory optimization method for large-scale model training according to claim 3, characterized in that: In step S3, the optimization objective function F is defined as minimizing the maximum difference D(G) of memory usage on devices at different stages, where G represents the computational graph partitioning of a model; a recursive relationship is set, and the model partitioning is gradually optimized by comparing the memory usage at different stages until a global optimal solution is reached; Through recursive calls, starting from the most basic partitioning unit, more complex model partitioning strategies are gradually built until the entire model is covered.

5. The pipeline parallel partitioning and memory optimization method for large-scale model training according to any one of claims 1 to 4, characterized in that: The method is scalable and can be combined with other distributed parallel techniques such as data parallelism (DP) and tensor parallelism (TP); and allows differentiated recomputation strategies to be set between different layers of the model according to specific performance and resource requirements.

6. The pipeline parallel partitioning and memory optimization method for large-scale model training according to any one of claims 1 to 5, characterized in that: Also includes: Analyze and adjust batch size based on memory usage and model partitioning strategy to improve training efficiency; By optimizing the recalculation strategy, the recalculation ratio in the back propagation in the later stage is reduced, and the additional calculation and time overhead are reduced.

7. The pipeline parallel partitioning and memory optimization method for large-scale model training according to any one of claims 1 to 6, characterized in that: Also includes: Identify substructures in the model, such as attention layers and feed-forward layers, and assess their isomorphism; The isomorphism of substructures is used to simplify the search space of dynamic programming algorithms, reduce calculation time and improve algorithm efficiency.

8. The pipeline parallel partitioning and memory optimization method for large-scale model training according to claim 7, characterized in that: The model includes a Transformer large model with attention layer and feed-forward layer substructures.

9. The pipeline parallel partitioning and memory optimization method for large-scale model training according to any one of claims 1 to 8, characterized in that: Also includes: Verify the effectiveness of the algorithm in an actual pipeline parallel environment so that the model partitioning strategy can actually improve training efficiency and memory utilization; tune the algorithm based on the verification results to adapt to different model sizes and training requirements.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the pipeline parallel partitioning and memory optimization method for large-scale model training as described in any one of claims 1 to 9.

Images