A pipeline parallel training method for multimodal large models

Abstract

The present invention discloses a pipeline parallel training method for a large multimodal model, and belongs to the field of model training technology. The method calculates the execution time and video memory occupancy data of each layer in the large multimodal model, and calculates the weight assigned to each layer of the large multimodal model based on the execution time and video memory occupancy data of each layer, thereby obtaining the initial stage division result; the improved taboo search method is used to optimize the initial stage division result to obtain the optimized solution, and the solution is used as the final division result of the large multimodal model, and the final division result is used to perform pipeline parallel training on the large multimodal model. The present invention utilizes video memory resources more efficiently and balances the computing load, significantly reducing the training time of the large multimodal model, and has stronger applicability, and can maintain a high training throughput in complex situations such as limited video memory.

Description

Technical Field

[0001] The present invention relates to model training technology, and in particular to a pipeline parallel training method for a multimodal large model. Background Art

[0002] Multimodal models, capable of integrating data from multiple modalities, such as images and text, have achieved remarkable results in the field of artificial intelligence. However, as the scale of models and the sequence of data increase, the demand for graphics memory and computing further increases. Efficient distributed training methods are key to achieving large-scale model training.

[0003] Currently, the most commonly used distributed training methods are data parallelism and model parallelism. Data parallelism alone cannot be used when the model parameter size exceeds the memory capacity of a single device. Model parallelism, which includes pipeline parallelism and tensor parallelism, is required. Pipeline parallelism works by splitting the model layers across different devices for computation. In the mainstream pipeline parallel scheduling method (1F1B), the device in the i-th stage of the pipeline (p stages) retains the intermediate computation results of pi micro-batches, resulting in an imbalance in memory usage and computational load across devices. This imbalance is particularly prominent for multimodal models composed of different modalities.

[0004] The pipeline parallel splitting technology of current mainstream frameworks and technologies is usually implemented based on 1F1B scheduling. The splitting algorithms of these methods mainly consider computing time or simply follow a parameter averaging strategy. The only option for recalculation is coarse-grained full recalculation, which will significantly increase training time. It is difficult to fully adapt to the differences in computing resources and video memory requirements between different modal sub-modules in multimodal model training. It does not consider the differences between different modal modules and cannot train the model efficiently. Summary of the Invention

[0005] Purpose of the invention: In response to the above problems, the purpose of the present invention is to provide a pipeline parallel training method for multimodal large models, determine the pipeline stage division of the model through the tabu search algorithm, apply the recalculation strategy based on the greedy method to each split point to determine the selection of pipeline splitting stage and recalculation layer, and achieve balanced optimization of computing load and video memory usage.

[0006] Technical solution: The present invention provides a pipeline parallel training method for a multimodal large model, comprising the following steps:

[0007] Calculate the execution time and memory usage of each layer in a large multimodal model;

[0008] Based on the execution time and memory usage data of each layer, the weight assigned to each layer of the multimodal large model is calculated. The accumulated value is calculated based on the weight and evenly distributed according to the number of devices to obtain the initial stage division result.

[0009] The improved tabu search method is used to optimize the initial stage division results to obtain the optimized solution, which is used as the final division result of the multimodal large model, and the final division result is used to perform pipeline parallel training on the multimodal large model.

[0010] Furthermore, the process of obtaining the initial stage division result includes:

[0011] Calculate the weight assigned to each layer of the multimodal model using the following formula:

[0012] ,

[0013] Where, is the weight of the kth layer, and Represent the adjustment coefficients of execution time and video memory usage data respectively, and ; Indicates the fixed memory usage of the kth layer, represents the total execution time of the kth layer;

[0014] Accumulate the weights of each layer in order , when the accumulated weight exceeds the total weight for the first time hour, , the first layer to the current layer are divided into the first stage , and so on, it is divided into p stages, recorded as , each stage is assigned a GPU device.

[0015] Furthermore, the steps of optimizing the initial stage partitioning result using the improved tabu search method include:

[0016] Step 301: Use the initial phase division result as the current solution, clear the taboo table, and set the maximum adjustment step size. ;

[0017] Step 302: Generate neighborhood solutions: For the current solution, use a step size of Adjust adjacent stage S i and S i+1 The segmentation point b generates a neighborhood solution set;

[0018] Step 303: For each domain solution in the neighborhood solution set, calculate the memory usage of each GPU device in the corresponding partitioning scheme; if it does not exceed the upper limit of the GPU device, jump to step 304; otherwise, use a greedy method to recalculate and optimize the neighborhood solution;

[0019] Step 304: Evaluate the solution quality and select the optimal solution based on cost;

[0020] Step 305, filter the solution: calculate the cost of the current neighborhood solution. If the cost of the current neighborhood solution is lower than the cost of the optimal solution, then take the current neighborhood solution as the new optimal solution and put the current neighborhood solution into the taboo table. If the cost of no neighborhood solution is lower than the cost of the current optimal solution, then select the new optimal solution according to the probability. Select the suboptimal solution and then put it into the taboo table; where P is a randomly generated value between 0 and 1, and T is the current temperature parameter;

[0021] Step 306: Update the taboo table: If no better solution is found after several consecutive iterations, trigger the diversification operation and increase the temperature T to , clear half of the taboo table and randomly generate an initial solution; otherwise, reduce the temperature to ;in, is the preset upper temperature limit, is the temperature rise coefficient, is the temperature reduction coefficient;

[0022] Step 307 , iterate steps 302 to 306 repeatedly until the termination condition is met, and obtain the optimized stage division scheme.

[0023] Furthermore, step 302 includes:

[0024] Update the step size according to the temperature T , using the updated step size , move the split point b to the right to , or move left to , generate a new partitioning scheme and add it to the neighborhood solution set;

[0025] The step length The update formula is:

[0026] .

[0027] Furthermore, in step 303, the steps of recalculating and optimizing the neighborhood solution using a greedy method include:

[0028] Calculate the memory benefit of each layer and recalculation cost , that is, the time for the k-th layer forward propagation. For the k-th layer, the memory gain Defined as the activation value memory saved after enabling recalculation, recalculation cost is the time to re-execute the forward propagation;

[0029] Calculate the ratio of memory benefit to recalculation cost using the formula: ;

[0030] Sort all layers in the current stage by MCR value from high to low, and give priority to recalculating layers with higher values;

[0031] Finally, the index list of recalculated layers at each stage is output.

[0032] Furthermore, in step 303, the video memory calculation formula is as follows:

[0033] ,

[0034] Where, and Stages The starting layer index and ending layer index, k is the stage The layer index in , satisfies , Indicates fixed video memory usage. For recalculation variables, 1 means recalculation is enabled, 0 means it is not enabled.

[0035] Furthermore, step 304 includes:

[0036] The cost function Cost is constructed with the goal of minimizing the maximum stage training time. The formula is:

[0037] ,

[0038] Where, For stage S i The total training time is composed of the basic calculation time and the additional time introduced by recalculation, and the formula is:

[0039] ,

[0040] Where, represents the total execution time of the kth layer;

[0041] Calculate the cost of each neighborhood solution and select the neighborhood solution corresponding to the lowest cost as the optimal solution.

[0042] Furthermore, in step 1, the formula for calculating the execution time of each layer is:

[0043] ,

[0044] Where, They represent the execution time of the kth layer of the multimodal large model during the back propagation process.

[0045] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0046] 1. This invention uses tabu search combined with simulated annealing algorithm to ensure a better balance between video memory usage and computing load through search optimization strategy, thereby improving overall training efficiency;

[0047] 2. The present invention dynamically selects layers that need to be recalculated based on video memory, giving priority to layers with high video memory returns and low computational costs. Compared with the coarse-grained full recalculation method, the fine-grained recalculation strategy of the present invention significantly reduces unnecessary computational overhead and effectively controls video memory occupancy, which not only improves training efficiency but also optimizes resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flowchart of a pipelined parallel training method for a multimodal large model;

[0049] Figure 2 Schematic diagram of the pipeline splitting framework;

[0050] Figure 3 The memory usage of the CLIP model at each stage with a batch size of 4;

[0051] Figure 4 The memory usage of the CLIP model at each stage with a batch size of 8;

[0052] Figure 5 The acceleration of the experimental model when the batch size is 4;

[0053] Figure 6 The speedup of the experimental model when the batch size is 8. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of this application more clear, this application is further described in detail below with reference to the accompanying drawings and embodiments.

[0055] The pipeline parallel training method of a multimodal large model described in this embodiment is as follows: Figure 1 As shown, the method includes the following steps:

[0056] Step 1: Calculate the execution time and memory usage of each layer in the multimodal large model.

[0057] During the preprocessing phase, the layers of the large multimodal model are analyzed to obtain the computation time and memory requirements of each layer, providing a foundation for subsequent pipeline splitting and memory optimization. First, a time analysis is performed to quantify the computational overhead of each pipeline stage. The profiling and hooking features of the framework tool can be used to measure the execution time of each layer in the large multimodal model and calculate the base computation time of each stage.

[0058] Furthermore, the formula for calculating the execution time of each layer is:

[0059] ,

[0060] Where, and They represent the execution time of the kth layer of the multimodal large model during the forward propagation and backpropagation processes, respectively.

[0061] Memory requirement analysis is used to evaluate the memory usage of each stage. The purpose of memory requirement analysis is to provide data support for subsequent pipeline splitting and recalculation strategies to ensure that the training process runs efficiently within the device's memory limitations.

[0062] Step 2: Calculate the weight assigned to each layer of the multimodal large model based on the execution time and memory usage data of each layer. Calculate the cumulative value based on the weight and evenly distribute it according to the number of devices to obtain the initial stage division result.

[0063] In the initialization phase of the algorithm, a reasonable partitioning scheme needs to be generated as the starting point for the search. For this purpose, a hybrid cost allocation strategy is designed. The layers of a large multimodal model have different computing time and memory requirements, so the weight assigned to each layer of the model is calculated by taking the weighted sum of computing and memory usage. Based on the calculated weights, the cumulative value is calculated and evenly distributed according to the number of devices, the layers divided at each stage are determined, and the initial partitioning results are generated. .

[0064] Furthermore, the process of obtaining the initial stage division result includes:

[0065] Calculate the weight assigned to each layer of the multimodal model using the following formula:

[0066] ,

[0067] Where, is the weight of the kth layer, and Represent the adjustment coefficients of execution time and video memory usage data respectively, and ; Represents the fixed memory usage of the k-th layer, which is composed of model parameters, model gradients, and optimizer parameters. represents the total execution time of the kth layer;

[0068] Accumulate the weights of each layer in order , when the accumulated weight exceeds the total weight for the first time hour, , the first layer to the current layer are divided into the first stage , and so on, it is divided into p stages, recorded as , each stage is assigned a GPU device.

[0069] Step 3: Use the improved tabu search method to optimize the initial stage division results to obtain the optimized solution, use the solution as the final division result of the multimodal large model, and use the final division result to perform pipeline parallel training on the multimodal large model.

[0070] To achieve efficient model splitting for training large multimodal models, an optimized tabu search method was chosen. Tabu search, through its search mechanism, can find optimized stage partitioning results within complex layer allocations. After the initial partitioning scheme is generated, the stage partitioning is assigned by searching for neighborhood solutions. Neighborhood solutions are new solutions obtained by adjusting the pipeline partitioning stages. To optimize the search space and avoid confining the search range to the local optimum, this embodiment incorporates the concept of simulated annealing into the tabu search, introducing a temperature parameter T to enhance the search range. The temperature controls the number of candidate solutions and the adjustment range. Higher temperatures increase the search range. This results in more candidate solutions, which is conducive to finding the global optimal solution and avoiding premature escaping from the local optimum. Lower temperatures reduce the number of candidate solutions, focusing on the local optimization of the current solution. Adjusting the split points between adjacent stages to generate new solutions increases the flexibility of tabu search and allows for finding more optimal partitioning schemes.

[0071] Furthermore, the steps of optimizing the initial stage partitioning result using the improved tabu search method include:

[0072] Step 301: Use the initial phase division result as the current solution, clear the taboo table, and set the maximum adjustment step size. ;

[0073] Step 302: Generate neighborhood solutions: For the current solution, use a step size of Adjust adjacent stage S i and S i+1 The segmentation point b generates a neighborhood solution set;

[0074] Step 303: For each domain solution in the neighborhood solution set, calculate the memory usage of each GPU device in the corresponding partitioning scheme. If it does not exceed the upper limit of the GPU device, jump to step 304; otherwise, use a greedy method to recalculate and optimize the neighborhood solution;

[0075] Step 304: Evaluate the solution quality and select the optimal solution based on cost;

[0076] Step 305, Screening solution: Calculate the cost of the current neighborhood solution. If the cost of the current neighborhood solution is lower than the cost of the optimal solution, then take the current neighborhood solution as the new optimal solution and put the current neighborhood solution into the taboo list; if the cost of no neighborhood solution is lower than the cost of the current optimal solution, then select the sub-optimal solution according to the probability P < T, and then put the sub-optimal solution into the taboo list; where P is a value randomly generated between 0 and 1, and T is the current temperature parameter.

[0077] Step 306, Update the taboo list: If no better solution is found after several consecutive iterations, trigger the diversification operation, raise the temperature T to , empty half of the taboo list, and randomly generate an initial solution; otherwise, reduce the temperature according to the exponential cooling strategy ; where is the preset upper limit of the temperature, is the temperature increase coefficient, such as 3.0; is the temperature decrease coefficient, such as 0.97;

[0078] Step 307, Repeat the iterative steps from Step 302 to Step 306 until the termination condition is met, and obtain the optimized phase division scheme.

[0079] To prevent searching for duplicate solutions and getting stuck in local loops due to repeated searches, a taboo list is used to record a previous segment of the executed solution. The initial value of the taboo list is set to , which is used to balance the previous search results and the exploration range, and dynamically adjusts the length of the taboo list according to the search status. If a better solution is found, it indicates that the current search direction is effective, and the length of the list is shortened to accelerate local optimization. On the contrary, if there is no better solution for a long time, it is necessary to jump out of the current area and increase the length of the list to expand the search range.

[0080] Furthermore, Step 302 includes:

[0081] Update the step size according to the temperature T , and use the updated step size , move the splitting point b to the right to , or move it to the left to , generate a new division scheme, and add it to the neighborhood solution set;

[0082] where the update formula for the step size is:

[0083] .

[0084] To ensure that the memory usage of each stage meets the device limit, this example uses a greedy method to perform recalculation optimization. This strategy is called before each evaluation of the candidate solution to dynamically adjust the recalculation layer allocation of each stage, thereby optimizing the memory usage and maintaining the balance of the computational load. By analyzing the memory benefit and recalculation cost of each layer, the algorithm can minimize the additional computational overhead and achieve efficient resource allocation while meeting the memory limit. If the initial memory If the device limit is not exceeded, recalculation is not required. Otherwise, analyze the recalculation value of each layer. For the kth layer, the memory benefit is defined as the activation value memory saved after recalculation is enabled. The activation value in 1F1B scheduling will increase with the stage depth. Zooming in, the recomputation cost is the time to re-execute the forward propagation, that is, the ratio of the computing memory benefit to the recomputation cost.

[0085] Furthermore, in step 303, the steps of recalculating and optimizing the neighborhood solution using a greedy method include:

[0086] Calculate the memory benefit of each layer and recalculation cost , that is, the time for the k-th layer forward propagation. For the k-th layer, the memory gain Defined as the activation value memory saved after recalculation is enabled. The activation value in 1F1B scheduling will change with the stage depth. Amplification, the formula is: , where is the size of the intermediate result of the kth layer, including the output tensor and the internal tensor, p represents the total number of pipeline stages, i represents the stage into which the kth layer is divided; recalculation cost is the time to re-execute the forward propagation;

[0087] Calculate the ratio of memory benefit to recalculation cost using the formula: ;

[0088] Sort all layers in the current stage by MCR value from high to low, and give priority to recalculating layers with higher values;

[0089] Finally, the recalculation layer index list of each stage is output. The recalculation layer index list specifies which layers need to be recalculated to generate intermediate results during training and used for video memory calculations.

[0090] Furthermore, in step 303, the video memory calculation formula is as follows:

[0091] ,

[0092] Where, and Stages The starting layer index and ending layer index, k is the stage The layer index in , satisfies , Indicates fixed memory usage, which is composed of model parameters, model gradients, and optimizer parameters. For recalculation variables, 1 means recalculation is enabled, 0 means it is not enabled.

[0093] After generating candidate solutions, their quality needs to be evaluated to select the optimal solution. Before evaluating the current solution, a memory-aware recalculation strategy must be invoked to ensure that the solution meets training constraints before evaluation. The optimization goal is to minimize the training iteration time. In pipelined 1F1B scheduling, the training time is determined by the computation time of the longest stage, meaning the optimization goal can be translated into minimizing the maximum computation time of each stage. The lowest-cost solution is then selected based on cost, and suboptimal solutions are accepted with a temperature-dependent probability to expand the search space and avoid premature convergence.

[0094] Furthermore, step 304 includes:

[0095] The cost function Cost is constructed with the goal of minimizing the maximum stage training time. The formula is:

[0096] ,

[0097] Where, For stage S i The total training time is composed of the basic calculation time and the additional time introduced by recalculation, and the formula is:

[0098] ,

[0099] Where, represents the total execution time of the kth layer;

[0100] Calculate the cost of each neighborhood solution and select the neighborhood solution corresponding to the lowest cost as the optimal solution.

[0101] After evaluating the cost of the solution, it enters the iterative optimization stage, and tries to find a better partitioning solution through continuous search. During the search process, the maximum number of iterations is set, which can not only ensure sufficient exploration of the solution space, but also limit the longest search time to speed up the algorithm. Then, termination conditions are introduced. That is, when no better solution is found after the number of iterations reaches the preset number, the search is terminated in advance to avoid ineffective search. At the beginning of the search, the temperature parameter is set to T. The T for finding the neighborhood solution previously is passed in here, and the initial value of T is set to 1.0 to control the entire iterative process. By continuously reducing the temperature value, the algorithm has the ability to explore a wide range of solution spaces in the initial stage, and gradually performs local search optimization as the iteration continues. Solutions are screened based on the cost calculation of the neighborhood solution. If the cost of the current solution is lower than the current best solution, the current partitioning scheme is updated, and the taboo table is updated according to the improvement amplitude. If there is no solution with a cost lower than the current one, the sub-optimal solution is accepted with a probability P < T. P is a randomly generated probability. This mechanism dynamically adjusts the acceptance rate using the temperature and the current value to ensure search diversity in the initial stage. To avoid being trapped in local optima, when the continuous iteration exceeds a certain number of times, a set diversity operation is triggered, the temperature is increased, half of the taboo table is cleared, and then the initial solution is generated. Through this iterative optimization mechanism, the algorithm can gradually approach the global optimal solution in the complex layer allocation of the multi-modal model, ensuring continuous improvement in the balance of computational load and video memory occupancy.

[0102] The splitting results can be evaluated, and the evaluation metrics include the single-iteration time, the peak video memory of each stage, the proportion of the computational time of each stage, and the GPU utilization rate. The single-iteration time is defined as the average time taken to complete one training batch; the peak video memory of each stage is the maximum video memory occupancy of each pipeline stage; the proportion of the computational time of each stage is the percentage of the computational time of this stage in the total training time. The GPU utilization rate is the average usage rate of the GPU during training. The calculation formula for each metric is as follows:

[0103] ,

[0104] ,

[0105] ,

[0106] where IT represents the single-iteration time, Total Time is the total duration of completing all batch iterations, is the total number of batches; represents the proportion of the computational time of the i-th stage, is the sum of the computational times of all stages, GPU Utilization represents the GPU utilization rate, Model FLOPs is the number of floating-point operations executed by the model, FLOPS is the number of floating-point operations per second of the GPU, and Time is the total training time.

[0107] Figure 2 This is the overall framework and optimization process for the parallel training of multimodal large model pipelines in the present invention, including the technical process in the upper part and the sample diagram in the lower part. The upper part starts with the allocation of models and hardware configurations, and finally outputs the optimized stage division scheme by analyzing performance data including computing time and video memory usage, pipeline split optimization and recalculation strategy; the lower part shows that after multimodal data is input, the multimodal model is divided into several stages, such as stage 0 to stage 3, and assigned to devices for model training.

[0108] This invention relates to the distributed training of large multimodal models. Specifically, it proposes an efficient pipelined parallel training method for large deep learning models that need to simultaneously process multiple modal data, such as images and text. This method is applied to the distributed training of large multimodal models on multi-GPU clusters, specifically models with over a billion parameters that simultaneously process multiple data types, such as images and text. The trained large multimodal models can then be used to process both images and text.

[0109] To further illustrate the effectiveness and superiority of the pipelined parallel training method for large multimodal models described in this paper, the following example is used. Two representative large multimodal models, CLIP and COCA, were selected as experimental models. These two experimental models were classified using the method described in this example. CLIP is an image-text retrieval model, while COCA is used for generative image-text tasks. The model size is approximately 20 billion parameters, and the specific parameter configurations are shown in Table 1 below.

[0110] Table 1

[0111]

[0112] This example is implemented on a cluster consisting of two machines, each equipped with eight H800 80GB graphics cards and two Intel(R) Xeon(R) Platinum 8468v CPUs. The GPUs are connected via NVLinks. The software environment uses CUDA 11.8 and PyTorch 2.4.1. PipeDream, a generalized pipeline parallelism system for deep neural network training, is used as a baseline. PipeDream uses dynamic programming to partition the model, and both an equal split and a full recalculation are compared. The iteration time and per-stage memory usage of the CLIP and COCA models are measured to evaluate memory utilization and speedup.

[0113] Depend on Figure 3 and Figure 4The figure shows the memory usage of each stage of the 19.7B CLIP model with batch sizes of 4 and 8. The black line is the GPU memory upper limit, and the stage exceeding the memory upper limit is the evaluation value. even is the parameter equalization provided by the distributed framework by default, even-ful is based on even plus full recalculation, PipeDream-ful is PipeDream plus full recalculation, and the method described in the present invention is denoted as Mpipe. It can be seen from the figure that the GPU memory utilization is significantly improved compared with the baseline. The single-node GPU utilization in the four batches is the highest, exceeding 95%. The memory utilization of the eight batches exceeds the baseline by more than 15GB on average. Since the early stages need to save more activation values, the memory of the early stages is significantly higher than that of the later stages.

[0114] Figures 5 and 6 The iteration time speedup ratios for the 19.7B CLIP and COCA models indicate that the method of the present invention can accelerate training time by up to 1.12 times and 1.17 times in different batches. In the figure, OOM indicates memory overrun. The method of the present invention combines computational cost with memory optimization to balance memory and computation, resulting in better computational efficiency than both averaging and PipeDream. This indicates that the iteration time of the method proposed in the present invention is superior to the baseline.

Claims

1. A pipeline parallel training method for a multimodal large model, characterized in that: The steps include: Calculate the execution time and memory usage of each layer in a large multimodal model; Based on the execution time and memory usage data of each layer, the weight assigned to each layer of the multimodal large model is calculated. The accumulated value is calculated based on the weight and evenly distributed according to the number of devices to obtain the initial stage division result. The improved tabu search method is used to optimize the initial phase partitioning results to obtain the optimized solution, which is used as the final partitioning result of the multimodal large model. The final partitioning result is used to perform pipeline parallel training on the multimodal large model. The process of obtaining the initial stage division results includes: Calculate the weight assigned to each layer of the multimodal model using the following formula: ， Where, is the weight of the kth layer, and Represent the adjustment coefficients of execution time and video memory usage data respectively, and ; Indicates the fixed memory usage of the kth layer, represents the total execution time of the kth layer; Accumulate the weights of each layer in order , when the accumulated weight exceeds the total weight for the first time hour, , the first layer to the current layer are divided into the first stage , and so on, it is divided into p stages, recorded as , each stage is assigned a GPU device; The steps of optimizing the initial phase partitioning results using the improved tabu search method include: Step 301: Use the initial phase division result as the current solution, clear the taboo table, and set the maximum adjustment step size. ; Step 302: Generate neighborhood solutions: For the current solution, use a step size of Adjust adjacent stage S i and S i+1 The segmentation point b generates a neighborhood solution set; Step 303: For each domain solution in the neighborhood solution set, calculate the memory usage of each GPU device in the corresponding partitioning scheme; if it does not exceed the upper limit of the GPU device, jump to step 304; otherwise, use a greedy method to recalculate and optimize the neighborhood solution; Step 304: Evaluate the solution quality and select the optimal solution based on cost; Step 305, filter the solution: calculate the cost of the current neighborhood solution. If the cost of the current neighborhood solution is lower than the cost of the optimal solution, then take the current neighborhood solution as the new optimal solution and put the current neighborhood solution into the taboo table. If the cost of no neighborhood solution is lower than the cost of the current optimal solution, then select the new optimal solution according to the probability. Select the suboptimal solution and then put it into the taboo table; where P is a randomly generated value between 0 and 1, and T is the current temperature parameter; Step 306: Update the taboo table: If no better solution is found after several consecutive iterations, trigger the diversification operation and increase the temperature T to , clear half of the taboo table and randomly generate an initial solution; otherwise, reduce the temperature to ;in, is the preset upper temperature limit, is the temperature rise coefficient, is the temperature reduction coefficient; Step 307 , iterate steps 302 to 306 repeatedly until the termination condition is met, and obtain the optimized stage division scheme.

2. The pipeline parallel training method for a multimodal large model according to claim 1, characterized in that: Step 302 includes: Update the step size according to the temperature T , using the updated step size , move the split point b to the right to , or move left to , generate a new partitioning scheme and add it to the neighborhood solution set; The step length The update formula is: 。 3. The pipeline parallel training method for a multimodal large model according to claim 2, characterized in that: In step 303, the steps of recalculating and optimizing the neighborhood solution using a greedy method include: Calculate the memory benefit of each layer and recalculation cost , that is, the time for the k-th layer forward propagation. For the k-th layer, the memory gain Defined as the activation value memory saved after enabling recalculation, recalculation cost is the time to re-execute the forward propagation; Calculate the ratio of memory benefit to recalculation cost using the formula: ; Sort all layers in the current stage by MCR value from high to low, and give priority to recalculating layers with higher values; Finally, the index list of recalculated layers at each stage is output.

4. The pipeline parallel training method for a multimodal large model according to claim 3, characterized in that: In step 303, the video memory calculation formula is as follows: ， Where, and Stages The starting layer index and ending layer index, k is the stage The layer index in , satisfies , Indicates fixed video memory usage. For recalculation variables, 1 means recalculation is enabled, 0 means it is not enabled.

5. The pipeline parallel training method for a multimodal large model according to claim 4, characterized in that: Step 304 includes: The cost function Cost is constructed with the goal of minimizing the maximum stage training time. The formula is: ， Where, For stage S i The total training time is composed of the basic calculation time and the additional time introduced by recalculation, and the formula is: ， Where, represents the total execution time of the kth layer; Calculate the cost of each neighborhood solution and select the neighborhood solution corresponding to the lowest cost as the optimal solution.

6. The pipeline parallel training method for a multimodal large model according to claim 5, characterized in that: In step 1, the formula for calculating the execution time of each layer is: ， Where, They represent the execution time of the kth layer of the multimodal large model during the back propagation process.

Images