Pipeline optimization method and system for accelerating lightweight training of multimodal large models

Abstract

This invention discloses a pipeline optimization method and system for accelerating lightweight training of large multimodal models. The method includes: acquiring a dataset and determining the batch size, and setting the number of pipeline stages and micro-batches based on the number of GPUs, requiring that the number of pipeline stages and micro-batches be an integer multiple of the number of GPUs; dividing the model to be trained into a number of pipeline stages according to its structure, with the computational cost of different parts differing by no more than 10%; deploying the model at different stages onto GPUs according to predefined rules; and performing the training process on each GPU according to pipeline parallelism rules. This invention leverages the similar time consumption of forward recursion and backward propagation in lightweight training of large multimodal models, reducing GPU memory usage while minimizing pipeline parallelism bubbles and improving training efficiency.

Description

Technical Field

[0001] This invention belongs to the field of multimodal large model training technology, specifically relating to a lightweight training acceleration method for multimodal large models. Background Technology

[0002] In recent years, with the continuous growth of computing resources and data scale, multimodal large models (MLLMs) have gradually become an important research direction in the field of artificial intelligence. These models achieve breakthroughs in complex tasks such as image and text understanding, image and text generation, and cross-modal retrieval by fusing information from multiple modalities, including images, text, and audio. In a typical architecture, a multimodal large model usually consists of an image encoder (such as Vision Transformer, ViT, Convolutional Neural Network, CNN), a modal projector (such as Multilayer Perceptron, MLP, linear layer), and a large language model (such as LLaMa, Qwen series). The modal projector maps different modal embeddings into a unified semantic space.

[0003] However, multimodal large-scale models have a huge number of parameters, resulting in extremely high demands on GPU memory and computing resources during training. In conventional training mode, the entire model participates in forward and backward propagation, consuming a large amount of GPU memory and easily causing memory overflow in multi-GPU cluster training. In addition, the large number of intermediate activation values ​​transferred during training further exacerbates the GPU memory pressure, seriously affecting the model's scalability and training efficiency.

[0004] To alleviate memory bottlenecks, pipeline parallelism is widely used in the training of large-scale neural networks. This technique divides the model into several stages and deploys different stages on different computing nodes, processing multiple training samples in a pipeline manner, thus distributing the memory pressure across each stage. Compared to data parallelism and tensor parallelism, pipeline parallelism has the advantages of low memory consumption and ease of deployment, and has become an important acceleration strategy for training ultra-large-scale models such as large language models.

[0005] However, applying pipelined parallelism to multimodal models has also revealed new problems. In actual training, especially in the lightweight training paradigm of "freezing the backbone model and training only the modal projectors," compared to traditional large language model training, this training paradigm does not update the parameters of the large language model part during backpropagation, and the computational cost of backpropagation is only half that of traditional training. Existing pipelined parallel methods are all designed under the premise that backpropagation takes twice as long as forward inference, and special optimizations have been made for pipeline scheduling, making actual deployment relatively complex.

[0006] In the prior art, patent CN118069375A discloses a pipeline parallel optimization method and apparatus for training large models in a data center. The method includes: obtaining the task list and task queue of each working device in the initial pipeline of the large model to be trained; obtaining the target task of at least one working device at the current time according to preset empirical rules; updating the corresponding task list according to the target task of at least one working device; having at least one working device execute the updated task list until the task queue is cleared, thus obtaining a completed task list; optimizing the initial pipeline to execute the pipeline parallelism of the large model to be trained in a preset data center. This invention can select the current task of a working device based on preset empirical rules to optimize the pipeline of the large model to be trained in real time, thereby further reducing bubble time during model training and inference, making it more efficient and economical. While using local data parallelism to reduce bubbles in the startup and cooling phases of forward propagation and backpropagation during model training reduces bubbles in these phases, it requires deploying two model phases on half of the devices, consuming more device memory. Furthermore, as the number of pipeline phases increases and the computational load of each phase decreases, the overall improvement in training speed is relatively small. Patent CN119250147A discloses an asynchronous multi-directional pipelined parallel method for accelerating large model training. The method includes: creating multiple pipelined parallel groups with different device orders between groups; sequentially allocating model slices (divided into segments of the large model to be trained) to each device in each group, with each group containing a complete large model to be trained; selecting any group as the optimizer registration parallel group and registering optimizers for the model slices of each device within it; training the large model to be trained according to an asynchronous multi-directional pipeline scheduling method, where each device accumulates the gradients of the same model slice on various micro-data batches, and when the accumulated micro-data batches constitute a data batch, immediately proceeding to the next step or at a pipeline vacancy position; and aggregating the accumulated gradients onto the device in the optimized parallel group and using the aggregated gradients for parameter updates. This invention can ensure high throughput while maintaining training convergence, reduce storage overhead without increasing additional communication overhead, and perform multiple pipelines on the same group of devices, with different device orders in each pipeline. The pipeline scheduling in this patent is quite complex, and each additional pipeline requires deploying a model stage on each device, consuming a significant amount of device memory. Furthermore, neither of the two related patents mentioned above specifically optimizes the lightweight training paradigm of freezing the backbone model and training only the modal projector. Summary of the Invention

[0007] To address the problems existing in the prior art, this invention provides a pipeline optimization method and system for accelerating lightweight training of multimodal large models. This method optimizes the training paradigm that only trains modal projectors, making it easy to implement in actual deployment and achieving higher efficiency and lower memory usage.

[0008] To achieve the above objectives, in a first aspect, the present invention provides a pipeline optimization method for accelerating lightweight training of multimodal large models, comprising the following steps: S1. Obtain the dataset and determine the batch size. Set the number of pipeline stages and the number of micro-batches to integer multiples of the number of GPUs based on the number of GPUs. The data in the dataset consists of images and text descriptions of those images. S2, divide the multimodal model to be trained into several parts according to its network structure, which is the same number as the pipeline stages; S3 deploys models at different stages onto GPUs according to predefined rules; S4 is trained on each GPU according to pipelined parallel rules.

[0009] Furthermore, in S3, deploying models at different stages onto the GPU according to predefined rules includes: It has One GPU, numbered as , Each stage of the production line is numbered as follows: Then the first The phase model is deployed to the first stage. One GPU, .

[0010] Furthermore, in S4, training on each GPU according to the pipeline parallelism rules includes: S401, constructs downlink and uplink steps on each GPU, and... Each micro-batch is divided into a group, the down-order sequence is the odd-numbered steps of each micro-batch, and the up-order sequence is the even-numbered steps of each micro-batch. S402 processes downlink and uplink steps on the GPU in a predefined order.

[0011] Furthermore, in S401, the downlink and uplink steps on each GPU are constructed, including: Training is decomposed into forward inference and backward propagation. Each GPU performs forward inference and backward propagation of each micro-batch of data in the order given by the rules for its model stages. Each GPU needs to process its own batch of micro-batch data. Each model stage Next forward recursion and In the second backpropagation, the model stages held by the GPU are set from front to back as follows: Define sequence ,in Indicates the first The phase model is forward recursively applied. Indicates the first If the phase model is backpropagated, the processing sequence on this GPU is: in Indicates proceeding to the first The first micro-batch One step, It is a downward sequence. Let be the upward sequence. Then we have:

[0012] .

[0013] Furthermore, in S402, when processing downlink and uplink steps on the GPU according to the set order: No. Each GPU performs After the next downstep, downstep-upstep steps are performed alternately, and finally the remaining upstep steps are cleared.

[0014] Furthermore, in S2, the difference in computational load between each stage does not exceed 10%.

[0015] Secondly, the present invention provides a lightweight training system for multimodal large models, including a dataset processing module, a partitioning module, a deployment module, and a training module; The dataset processing module is used to acquire the dataset and determine the batch size, and set the number of pipeline stages and the number of micro-batches to integer multiples of the number of GPUs based on the number of GPUs. The data in the dataset consists of images and text descriptions of those images. The partitioning module is used to divide the multimodal model to be trained into several parts according to its network structure, which is the same number as the pipeline stages. The deployment module is used to deploy models at different stages onto the GPU according to predefined rules; The training module is used to train on each GPU according to the pipeline parallel rules.

[0016] Thirdly, the present invention also provides a computer program product containing instructions that, when the instructions are executed on a computer, cause the computer to perform the above-described pipeline optimization method for accelerating lightweight training of multimodal large models.

[0017] Fourthly, the present invention provides a computer device, including a processor and a memory, wherein the memory is used to store a computer executable program, the processor reads part or all of the computer executable program from the memory and executes it, and the processor can realize the above-mentioned pipeline optimization method for accelerating lightweight training of multimodal large models when executing part or all of the computer executable program.

[0018] Finally, a computer-readable storage medium may be provided, which stores a computer program that, when executed by a processor, enables the above-mentioned pipeline optimization method for accelerating lightweight training of multimodal large models.

[0019] Compared with the prior art, the present invention has at least the following beneficial effects: The pipeline optimization method for accelerating lightweight training of multimodal large models provided in this application differs from the pipeline parallel method used for traditional large language model training. It aims to improve training efficiency and reduce memory usage. It is specifically optimized for the training paradigm of "freezing the modal encoder and the large language model and training only the modal projector". It utilizes the fact that the forward recursion and backpropagation in this training paradigm have similar time consumption for pipeline scheduling, thereby improving training efficiency and making the actual deployment process simpler. Attached Figure Description

[0020] Figure 1 The flowchart shows the pipeline optimization method for accelerating lightweight training of multimodal large models provided in this embodiment.

[0021] Figure 2 The pipeline optimization method for accelerating lightweight training of multimodal large models provided in this implementation uses a pipeline scheduling graph where the number of micro-batches is 3 times the number of GPUs and the number of micro-batches is 1 times that of the pipeline. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, exemplary in nature, and intended to explain this invention, and should not be construed as limiting this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort should fall within the scope of protection of this invention.

[0023] refer to Figure 1 The present invention provides a pipeline optimization method for accelerating lightweight training of multimodal large models, comprising the following steps: S1, Dataset and Training Parameter Settings. In this embodiment, the dataset required for multimodal training is first obtained, and training-related parameters are set, including batch size, number of pipeline stages, and number of micro-batches. To fully utilize hardware resources, the number of pipeline stages and the number of micro-batches must be integer multiples of the number of GPUs, thereby ensuring that tasks are evenly distributed among the GPUs and improving parallel efficiency.

[0024] The dataset contains images and their corresponding text descriptions to support image-text alignment training. As a specific example, the COCO Caption dataset is used for experiments, and training is conducted on four GPUs. To achieve high pipeline utilization, the number of pipeline stages is set to 12 and the number of micro-batches is set to 4 in this embodiment. Figure 2 As shown.

[0025] S2, Model Segmentation and Load Balancing. After determining the number of pipeline stages, the multimodal model to be trained is divided into several parts according to its network structure, the same number as the number of pipeline stages. The computational load of each stage should be as similar as possible to avoid uneven GPU idleness or load, which would lead to a decrease in overall efficiency. To ensure computational balance, the difference in computational load between stages should not exceed 10%.

[0026] In this embodiment, the total number of parameters in the model to be trained is 12B (i.e., 12 billion parameters). Based on 12 pipeline stages, the model is divided into 12 sub-modules, each with approximately 1B of parameters, and these sub-modules are deployed to different GPUs. This partitioning method helps to achieve high computational parallelism and memory utilization while ensuring model integrity.

[0027] S3 deploys models at different stages onto the GPU according to predefined rules. Specifically, it sets... One GPU, numbered as , Each stage of the production line is numbered as follows: Then the first The phase model was deployed to the first stage. GPUs, of which That is, to divide the model stage according to " "Deployed to the GPU in a backflip manner. In this step, such as..." Figure 2 As shown, stages 1 to 4 of the model are deployed sequentially on GPUs 1 to 4. Stages 5 to 8 of the model are reversed, starting from GPU 4 and being deployed sequentially on GPU 1. Stages 9 to 12 are reversed again, starting from GPU 1 and being deployed to GPU 4.

[0028] S4 performs the training process on each GPU according to the pipeline parallelism rules. Specifically, it includes: S401 constructs the downlink and uplink steps on each GPU. The training process is decomposed into two phases: Forward Pass and Backward Pass. Each GPU is only responsible for the computational tasks of its assigned model phase, and performs alternating forward and backward processing on multiple micro-batch data according to rules. Assume there are a total of... Each micro-batch holds [amount] for each GPU. Each model stage requires each GPU to execute Sub-forward reasoning and Second backpropagation. The model stages held by the GPU from front to back are set as follows: Define sequence ,in Indicates the first The phase model is forward recursively applied. Indicates the first Backpropagation of the phase model. The processing sequence on this GPU is then: in Indicates proceeding to the first The first micro-batch Each step (forward reasoning or backward propagation) It is a downward sequence. This is an upward sequence. That is, each... Each micro-batch is grouped into a set, with the down-row sequence consisting of the odd-numbered steps of each micro-batch and the up-row sequence consisting of the even-numbered steps of each micro-batch. Let... Then we have:

[0029] .

[0030] Each GPU holds 3 model stages and 4 micro-batches, each micro-batch requiring 6 steps. Micro-batches are grouped into sets of 4; the downlink steps are steps 1, 3, and 5 of each micro-batch, and the uplink steps are steps 2, 4, and 6 of each micro-batch. Let... Then we have:

[0031] That is, on each GPU, the downlink steps are the first and third forward inferences and the second backpropagation for each micro-batch, and the uplink steps are the second forward inference and the first and third backpropagation for each micro-batch.

[0032] S402 processes the downlink and uplink steps on the GPU according to a predefined sequence. Based on the defined pipeline rules, each GPU must follow a specific order and staggered relationship when executing training tasks. Specifically, the... Each GPU needs to execute at startup. The next downstep step fills the pipeline; subsequently, downstep and upstep steps alternate, and finally, the remaining upstep steps are cleared at the end of the pipeline to ensure correct gradient backpropagation. Figure 2 For example, the first GPU performs 3 downlink steps, then alternates to perform 9 downlink-uplink steps, and finally performs 3 uplink steps; the second GPU performs 2 downlink steps, then alternates to perform 10 downlink-uplink steps, and finally performs 2 uplink steps; the third GPU performs 1 downlink step, then alternates to perform 11 downlink-uplink steps, and finally performs 1 uplink step; the fourth GPU directly performs 12 downlink-uplink steps.

[0033] In summary, the above-mentioned scheme provided by the embodiment of the pipeline optimization method for accelerating lightweight training of multimodal large models of the present invention obtains the dataset and determines the batch size, and sets the number of pipeline stages and micro-batch numbers according to the number of GPUs, requiring that the number of pipeline stages and micro-batch numbers be integer multiples of the number of GPUs; the model to be trained is divided into a number of pipeline stages according to its structure, with the computational cost of different parts differing by no more than 10%; the models of different stages are deployed to GPUs according to predefined rules; the training process is performed on each GPU according to the pipeline parallel rules; by specifically optimizing the training paradigm that only trains modal projectors, it is made easy to implement in deployment, and the training efficiency is improved and the memory usage is reduced. With the goal of improving training efficiency and reducing memory usage, the pipeline scheduling is performed by taking advantage of the similar time consumption of forward recursion and backpropagation in this training paradigm, ultimately achieving improved training efficiency and simplifying the actual deployment process.

[0034] Based on the technical concept of the method, the present invention also provides a lightweight training system for multimodal large models, including a dataset processing module, a partitioning module, a deployment module and a training module; The dataset processing module is used to acquire the dataset and determine the batch size, and set the number of pipeline stages and the number of micro-batches to integer multiples of the number of GPUs based on the number of GPUs. The data in the dataset consists of images and text descriptions of those images. The partitioning module is used to divide the multimodal model to be trained into several parts according to its network structure, which is the same number as the pipeline stages. The deployment module is used to deploy models at different stages onto the GPU according to predefined rules; The training module is used to train on each GPU according to the pipeline parallel rules.

[0035] On the other hand, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the pipeline optimization method for accelerating lightweight training of multimodal large models as described in the present invention.

[0036] The present invention can also provide a computer device, including a processor and a memory, wherein the memory is used to store a computer executable program, the processor reads the computer executable program from the memory and executes it, and the processor can implement the pipeline optimization method for accelerating lightweight training of multimodal large models described in the present invention when executing the computer executable program.

[0037] The computer program product includes computer instructions that, when loaded and executed on a computer, perform the methods described in the embodiments of this application. The computer may be a general-purpose computer, a special-purpose computer, or other programmable device. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means.

[0038] The computer device may be a laptop, a desktop computer, or a workstation.

[0039] The processor can be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or an off-the-shelf programmable gate array (FPGA).

[0040] The memory described in this invention can be an internal storage unit of a laptop, desktop computer, or workstation, such as memory or hard disk; or it can be an external storage unit, such as a portable hard disk or flash memory card.

[0041] Computer-readable storage media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media can include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. Random access memory can include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0042] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A pipeline optimization method for accelerating multi-modal large model lightweight training, characterized in that, Includes the following steps: S1. Obtain the dataset and determine the batch size. Set the number of pipeline stages and the number of micro-batches to integer multiples of the number of GPUs based on the number of GPUs. The data in the dataset consists of images and text descriptions of those images. S2, divide the multimodal model to be trained into several parts according to its network structure, which is the same number as the pipeline stages; S3 deploys models at different stages onto GPUs according to predefined rules; S4, trained on each GPU according to pipeline parallelism rules, specifically includes: S401, constructs downlink and uplink steps on each GPU, and... Each micro-batch is divided into a group, the down-order sequence is the odd-numbered steps of each micro-batch, and the up-order sequence is the even-numbered steps of each micro-batch. S402 processes downlink and uplink steps on the GPU in a predefined order; In S401, the downlink and uplink steps on each GPU are constructed, including: Training is decomposed into forward inference and backward propagation. Each GPU performs forward inference and backward propagation of each micro-batch of data in the order given by the rules for its model stages. Each GPU needs to process its own batch of micro-batch data. Each model stage Next forward recursion and In the second backpropagation, the model stages held by the GPU are set from front to back as follows: Define sequence ,in Indicates the first The phase model is forward recursively applied. Indicates the first If the phase model is backpropagated, the processing sequence on this GPU is: in, Indicates proceeding to the first The first micro-batch One step, It is a downward sequence. Let be the upward sequence. Then we have: 。 2. The pipeline optimization method for accelerating lightweight training of multimodal large models according to claim 1, characterized in that, In S3, deploying models at different stages onto the GPU according to predefined rules includes: It has One GPU, numbered as , Each stage of the production line is numbered as follows: Then the first The phase model is deployed to the first stage. One GPU, .

3. The pipeline optimization method for accelerating lightweight training of multimodal large models according to claim 1, characterized in that, When processing downlink and uplink steps on the GPU in the S402 according to a set order: No. Each GPU performs After the next downstep, downstep-upstep steps are performed alternately, and finally the remaining upstep steps are cleared.

4. The pipeline optimization method for accelerating lightweight training of multimodal large models according to claim 1, characterized in that, In S2, the difference in computational load between each stage does not exceed 10%.

5. A lightweight training system for multimodal large models, characterized in that, The pipeline optimization method for accelerating lightweight training of multimodal large models as described in any one of claims 1-4 includes a dataset processing module, a partitioning module, a deployment module, and a training module. The dataset processing module is used to acquire the dataset and determine the batch size, and set the number of pipeline stages and the number of micro-batches to integer multiples of the number of GPUs based on the number of GPUs. The data in the dataset consists of images and text descriptions of those images. The partitioning module is used to divide the multimodal model to be trained into several parts according to its network structure, the same number as the pipeline stages. The computational cost between each stage does not exceed 10%. The deployment module is used to deploy models at different stages onto the GPU according to predefined rules; The training module is used to train on each GPU according to the pipeline parallel rules.

6. A computer program product containing instructions, characterized in that, When the instructions are executed on a computer, the computer causes the computer to perform claim 1. Pipeline optimization method for accelerating lightweight training of multimodal large models as described in any of the four points.

7. A computer device, characterized in that, It includes a processor and a memory, the memory being used to store a computer-executable program, the processor reading part or all of the computer-executable program from the memory and executing it, and the processor executing part or all of the compute executable program is able to implement the pipeline optimization method for accelerating lightweight training of multimodal large models as described in any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program that, when executed by a processor, enables the pipeline optimization method for accelerating lightweight training of multimodal large models as described in any one of claims 1-4.

Images