An Efficient Hybrid Parallel Training Method for Multimodal Models

Abstract

The present invention discloses an efficient hybrid parallel training method for multi-modal models, belonging to the field of computer technology. The method first divides the multi-modal model architecture into multiple independent modal sub-modules according to the multi-modal model architecture, and each sub-module processes one type of modal data; then, by statically analyzing the computational load of each sub-module, a computing power allocation scheme is formulated; analyze the computational time and the number of parameters of each layer of the modal sub-module, and then use the partitioning algorithm to divide these layers into multiple stages, and combine asynchronous pipelining parallelism and data parallelism to accelerate the training. The present invention significantly improves the utilization rate of hardware resources and realizes the acceleration of the training process of the multi-modal model through the computing power allocation and hybrid parallel strategy for modal computational load.

Description

Technical Field

[0001] The present invention relates to a multi-modal model training method, specifically an efficient hybrid parallel training method for multi-modal models, belonging to the field of computer technology. Background Art

[0002] Multi-modal models provide a comprehensive perspective for complex tasks by jointly learning data from different modalities, significantly enhancing the model's reasoning ability. However, multi-modal models consist of multiple sub-modules with different functions, and these sub-modules have significant differences in structure, computational requirements, and data processing methods. For example, some modal sub-modules use deep convolutional neural networks or Vision Transformers to process high-dimensional data, with a heavy computational load, while some modal sub-modules process serialized language information based on the Transformer architecture, with relatively lighter computational requirements. This heterogeneity poses many challenges to existing distributed training strategies.

[0003] Existing distributed training strategies mainly include data parallelism and pipeline parallelism. Data parallelism divides the training data into sub-batches and distributes them to different devices for parallel processing. Each device holds a complete model copy and updates the parameters through gradient synchronization. However, in multi-modal models, due to the computational load differences between different modalities, a simple gradient synchronization method causes some devices to wait too long, resulting in waste of computational resources. Pipeline parallelism divides the model into multiple stages and distributes them to different devices for processing, reducing the memory burden of a single device. However, in multi-modal models, the uneven computational load of sub-modules easily forms a performance bottleneck, and some stages are idle waiting for forward computation, reducing the training efficiency. In addition, although asynchronous pipeline parallelism can improve throughput, it may lead to a decrease in accuracy due to outdated gradients, especially in the initial stage of training, which has a more obvious impact.

[0004] Existing technologies have problems such as uneven load, low training efficiency, and decreased accuracy in multi-modal model training. There is an urgent need for an efficient training method that can optimize for the heterogeneity of multi-modal models. Summary of the Invention

[0005] Object of the Invention: Aiming at the above problems, the object of the present invention is to provide an efficient hybrid parallel training method for multi-modal models, which significantly improves the utilization rate of hardware resources through computing power allocation and hybrid parallel strategies for modal computational loads.

[0006] Technical Solution: An efficient hybrid parallel training method for multi-modal models of the present invention includes the following steps:

[0007] Obtain a multi-modal model and a dataset required for training;

[0008] Perform architecture analysis on the multimodal model. According to the modal type of the input data, divide the multimodal model into multiple independent modal sub-modules, and each modal sub-module is dedicated to processing a specific modal data;

[0009] By statically analyzing the parameters of each modal sub-module, evaluate the computational load of each modal sub-module when processing global batch size data;

[0010] Based on the computational load analysis results, formulate a computing power resource allocation plan to determine the number of devices required for each modal sub-module;

[0011] According to the computing power resource allocation plan, adopt a partitioning algorithm for each modal sub-module to divide the modal sub-module into multiple consecutive pipeline stages;

[0012] Divide the training process into a warm-up stage and an acceleration stage, and use the data set to train the multimodal model using data parallelism and asynchronous pipeline parallelism respectively.

[0013] Further, the steps of formulating a computing power resource allocation plan based on the computational load analysis results to determine the number of devices required for each modal sub-module include:

[0014] Adopt the proportional allocation principle, and allocate the corresponding number of devices according to the computational load ratio of each modal sub-module and the total number of devices.

[0015] Further, the steps of adopting a partitioning algorithm for each modal sub-module according to the computing power resource allocation plan to divide the modal sub-module into multiple consecutive pipeline stages include:

[0016] Take each modal sub-module independently as the input of the partitioning algorithm, analyze the number of parameters of each layer of the modal sub-module, use dynamic programming to determine the optimal partitioning boundary according to the results of the hierarchical analysis, and divide the modal sub-module into multiple consecutive pipeline stages according to the optimal partitioning boundary to ensure the computational load balance of each pipeline stage.

[0017] Further, the steps of dividing the training process into a warm-up stage and an acceleration stage, and using the data set to train the multimodal model using data parallelism and asynchronous pipeline parallelism respectively include:

[0018] In the first 10% stage of the model training, adopt the data parallel strategy for training, use the linear warm-up learning rate, and when switching stages, the learning rate reaches the preset maximum value. This stage is recorded as the warm-up stage;

[0019] The remaining training stage is used as the acceleration stage, switch to the asynchronous pipeline parallel strategy, allocate the multiple pipeline stages divided by the multimodal model to their respective devices for asynchronous update, and at the same time allocate the training data set to each modal sub-module.

[0020] Furthermore, in the acceleration stage, each pipeline stage runs independently on its respective GPU and performs asynchronous forward computation and backpropagation. Meanwhile, the learning rate enters the decay stage, and a smooth decay curve of the learning rate is presented through a cosine function, finally decaying to near 0. Gradient accumulation is introduced, and the computation results of each pipeline stage are temporarily stored in the local buffer. After the gradients of a preset number of micro-batches are accumulated on the GPU in the pipeline, the gradients are updated uniformly.

[0021] Beneficial effects: Compared with the prior art, the remarkable advantages of the present invention are:

[0022] Aiming at the heterogeneity of multi-modal models, the present invention overcomes the deficiencies of existing methods in load balancing, training efficiency, and accuracy maintenance through computing power allocation and hybrid parallel strategies. Specifically, the present invention avoids the problems of excessive device waiting time and performance bottlenecks by allocating computing power resources according to the computational load, separating modalities, and dividing pipeline stages, thereby significantly improving the utilization rate of hardware resources; adopting asynchronous pipeline parallelism and gradient accumulation reduces the waiting time, accelerates the training process, and shortens the overall training time; at the same time, data parallelism and linear warm-up learning rate are adopted in the warm-up stage, and gradient accumulation and learning rate decay are introduced in the acceleration stage, effectively reducing the accuracy loss caused by asynchronous updates and ensuring the model performance. The hybrid parallel strategy adopted by the present invention means that the training process can be flexibly adjusted according to specific task requirements and hardware resources. Whether it is a small device or a large-scale cluster, this adaptability can ensure the efficient operation of the training process, and at the same time improve the scalability of the method, making it applicable to application scenarios of different scales. In summary, the present invention can significantly improve the utilization rate of hardware resources while accelerating the training process of multi-modal models, ensuring the training accuracy, and also having excellent scalability, which is an important improvement to the existing distributed training strategy. Brief Description of the Drawings

[0023] Figure 1 It is a flowchart of an efficient hybrid parallel training method for multi-modal models;

[0024] Figure 2 It is a schematic diagram of modality separation of a two-stream multi-modal model;

[0025] Figure 3 It is a schematic diagram of using data parallelism for training in the warm-up stage;

[0026] Figure 4 It is a schematic diagram of using asynchronous pipeline parallel training in the acceleration stage;

[0027] Figure 5 It is a comparison chart of the iteration time of the method described in the present invention with that of mainstream training strategies after normalizing the single-iteration time of the method described in the present invention to 1;

[0028] Figure 6 This is a comparison chart of the training accuracy of the method of the present invention with asynchronous pipelines and synchronous pipelines. Detailed implementation manners

[0029] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0030] An efficient hybrid parallel training method for a multimodal model according to this embodiment has a flowchart as Figure 1 shown, and this method includes the following steps:

[0031] Step 1: Obtain a multimodal model and a data set required for training.

[0032] For a given multimodal model, the multimodal model includes at least two of modalities such as text, speech, and image. Obtain a data set required for training. If the training model is a three-stream multimodal model of speech-text-image, then it is necessary to obtain a data set in which text, speech, and image correspond to each other.

[0033] Step 2: Perform architecture analysis on the multimodal model, and divide the multimodal model into multiple independent modal sub-modules according to the modal type of the input data. Each modal sub-module is dedicated to processing a specific modal data.

[0034] Combined with Figure 2 shown, in an example, the modal type of the input data of the multimodal model can be any two or three of text, speech, and image. If there are two, they are respectively denoted as modality 1 and modality 2. Then the given multimodal model is divided into 2 independent modal sub-modules, denoted as sub-module 1 and sub-module 2, and each modal sub-module processes one modal data.

[0035] Step 3: Evaluate the computational load of each modal sub-module when processing global batch size data by statically analyzing the parameters of each modal sub-module.

[0036] In the example, by analyzing the model structure and parameters of each modal sub-module, the number of floating-point operations (FLOPs) when processing a single sample is calculated, which is used as the main indicator of the computational load. According to the type of modal data processed, the dual-stream multimodal large model used is divided into modal sub-module 1 and modal sub-module 2. For modal sub-module 1 such as VisionTransformer as a multimodal model, FLOPs mainly come from the calculation of the self-attention mechanism and the feedforward network, which can be estimated by parameters such as the number of attention heads, sequence length, and embedding dimension. For modal sub-module 2 such as Transformer, FLOPs are concentrated in word embedding and attention calculations, which can be calculated based on the sequence length and number of layers. The calculation formulas for the number of floating-point operations of the convolutional layer, the fully connected layer, and the attention mechanism are as follows:

[0037] ,

[0038] Where, Represents the number of floating-point operations of the convolutional layer, where and Represent the height and width of the output feature map respectively, is the number of input channels, is the number of output channels, is the height and width of the convolution kernel, and the coefficient 2 means that each convolution operation contains one addition and multiplication. The convolution layer is the core part of the computational intensiveness.

[0039] ,

[0040] Where, is the number of floating point operations of the fully connected layer. In the Transformer model, the FLOPs calculation of the attention mechanism is more complicated. is the number of input neurons, is the number of output neurons, and the factor 2 also comes from multiplication and addition.

[0041] ,

[0042] Where, represents the number of floating point operations of the attention mechanism, is the sequence length, is the model dimension.

[0043] Based on the computational load (i.e., the number of floating-point operations per module 1 and 2), a computing resource allocation plan is developed. Typically, submodules with higher computational load and memory requirements are allocated more computing resources, while submodules with lower loads are allocated fewer resources. This approach provides a reasonable starting point for the training process.

[0044] Step 4: Based on the computational load analysis results, formulate a computing power resource allocation plan to determine the number of devices required for each modal sub-module.

[0045] According to the results of the computational load, formulate a computing power resource allocation plan. Generally, sub-modules with larger computational loads and memory requirements will receive more computing resource support, while sub-modalities with smaller loads will be allocated fewer resources. This plan provides a reasonable starting point for the training process.

[0046] Furthermore, the steps of formulating a computing power resource allocation plan based on the computational load analysis results to determine the number of devices required for each modal sub-module include:

[0047] Adopt the proportional allocation principle and allocate the corresponding number of devices according to the computational load ratio of each modal sub-module and the total number of devices.

[0048] In one example, if the calculated computational load ratio of sub-module 1 to sub-module 2 is 2:3 and the total number of GPU devices is 20, then sub-module 1 is allocated 8 GPU devices and sub-module 2 is allocated 12 GPU devices.

[0049] Step 5: According to the computing power resource allocation plan, use a partitioning algorithm for each modal sub-module to divide the modal sub-module into multiple consecutive pipeline stages.

[0050] Furthermore, the steps of using a partitioning algorithm for each modal sub-module according to the computing power resource allocation plan to divide the modal sub-module into multiple consecutive pipeline stages include:

[0051] Take each modal sub-module independently as the input of the partitioning algorithm, analyze the number of parameters in each layer of the modal sub-module, and use dynamic programming to determine the optimal partitioning boundary according to the results of the hierarchical analysis. Assume that the modal sub-module has n layers and is divided into k pipeline stages. The computational load of each layer is obtained through hierarchical analysis, and the formula is as follows:

[0052] ,

[0053] In the formula, i represents the first i layers of the modal sub-module, starting from 1, indicating that dynamic programming considers sub-problems from 1 to the first i layers; j represents the number of divided stages, which cannot exceed the maximum allowed number of stages k and is at least 1; m represents that the last stage starts from the m-th layer until the i-th layer; is the computational load of the last stage, that is, from the m-th layer to the i-th layer; is the minimum bottleneck load when the first m - 1 layers are divided into j - 1 stages.

[0054] By enumerating all possible m, the partitioning algorithm selects the one that makes the bottleneck load of the current partition The smallest solution, finally gives the minimum bottleneck load when dividing all n layers into k stages. By backtracking the dp table, the specific partition boundaries can be determined, and according to the optimal partition boundaries, the modal sub-modules are divided into multiple consecutive pipeline stages to ensure the balanced computational load of each pipeline stage.

[0055] In the example, k GPU devices are allocated to sub-module 1. Taking this sub-module 1 as the input of the partitioning algorithm, the number of parameters of each layer of this sub-module 1 is analyzed using the analysis algorithm, and then the modal sub-module is divided into k consecutive stages with balanced computational load using the partitioning algorithm.

[0056] Based on the computing power allocation results, the partitioning algorithm is used for each separated modality alone. At the same time, some lightweight modules may be combined to ensure that each sub-modal module can obtain a suitable model partition and ensure that the computational load of each stage is as balanced as possible. Taking the two-stream multi-modal model as an example, modalities with high computational load may be allocated separately, while modalities with low computational load are combined with other lightweight modules, thus avoiding a certain device from slowing down the overall training progress due to overloading. After the sub-module is divided into consecutive stages, data transmission is only performed at the stage boundaries, avoiding frequent global communication.

[0057] Step 6, divide the training process into a warm-up stage and an acceleration stage, and use the dataset to train the multi-modal model using data parallelism and asynchronous pipeline parallelism respectively.

[0058] In the early stage of training, due to the large span of parameter updates, the accuracy loss caused by asynchronous updates is more significant. As the model parameters gradually stabilize, the impact of asynchronous updates on accuracy will weaken. Therefore, in this example, the entire training process is divided into two stages: warm-up and acceleration, and different parallel configurations are used in different stages to optimize efficiency and accuracy.

[0059] Furthermore, the steps of dividing the training process into a warm-up stage and an acceleration stage, and using the dataset to train the multi-modal model using data parallelism and asynchronous pipeline parallelism respectively include:

[0060] Combined with Figure 3 , in the first 10% of the stages when the model starts training, for example, if 200 training epochs are set, the first 20 epochs are the warm-up stage, and the data parallelism strategy is used for training. A linear warm-up learning rate is used, and at the stage switch, the learning rate reaches the preset maximum value. This stage is recorded as the warm-up stage;

[0061] In the warm-up stage, the multi-modal model is regarded as a whole and data parallelism is used to ensure the stability of parameter updates. In this stage, by reducing the impact of gradient obsolescence, the initial improvement of model accuracy is guaranteed.

[0062] Except for the warm-up stage, the remaining training stage is taken as the acceleration stage. Switch to the asynchronous pipeline parallel strategy, allocate the multiple pipeline stages divided by the multi-modal model to their respective devices for asynchronous update, and at the same time allocate the training dataset to each modal sub-module.

[0063] When the model parameters gradually stabilize, switch to asynchronous pipeline parallelism and introduce the gradient accumulation technique to maximize the training throughput, thereby accelerating the overall training process.

[0064] Combining asynchronous pipeline parallelism and the stage training strategy can not only improve the training efficiency but also effectively alleviate the problem of accuracy degradation caused by asynchronous pipeline parallelism. This method makes full use of hardware resources and provides an efficient and flexible solution for the distributed training of multi-modal models.

[0065] Furthermore, in the acceleration stage, each pipeline stage runs independently on its respective GPU and performs asynchronous forward calculation and backward propagation. At the same time, the learning rate enters the decay stage, and the learning rate shows a smooth decay curve through the cosine function, and finally smoothly decays to close to 0. And introduce gradient accumulation. The calculation results of each pipeline stage are temporarily stored in the local buffer. When the gradients of the preset number of micro-batches are accumulated on the GPU in the pipeline, the gradient update is performed uniformly.

[0066] After completing the model partitioning, introduce the asynchronous pipeline parallel strategy in the acceleration stage to accelerate the training process. This strategy allows each computing device to process tasks in different stages without global synchronization, thereby maximizing device utilization and reducing idle time. The training data batch is split into multiple micro-batches and injected into the pipeline in sequence. When each device processes the current micro-batch, it can asynchronously pass the output of the previous micro-batch to the downstream device and start processing the next micro-batch at the same time. This effectively overlaps the computing and communication time and hides the communication latency. In the backward propagation, the gradient calculation and parameter update are performed asynchronously. The device updates the local parameters immediately after completing the gradient calculation of a micro-batch without waiting for other devices to synchronize, further reducing the synchronization waiting time. Through this asynchronous mechanism, this strategy can achieve efficient multi-device collaboration in forward and backward propagation, make full use of computing resources, and significantly accelerate the overall training.

[0067] When the warm-up stage ends, switch to the acceleration stage, as Figure 4 Taking the two-stream multi-modal model as an example, according to the allocation results of each modal sub-module, divide sub-module 1 and sub-module 2 into multiple pipeline stages, use asynchronous pipeline parallelism, and introduce the gradient accumulation technique to mitigate the impact of accuracy loss caused by weight obsolescence. In the asynchronous pipeline stage, use a smaller learning rate, and the expression is:

[0068] ,

[0069] where \(l_r\) is the learning rate for this round, is the preset learning rate, , and are the current step, the switching step, and the total step respectively. The learning rate shows a smooth decay curve through the cosine function and finally decays smoothly to approach 0. A smaller learning rate can reduce the impact of weight obsolescence caused by too large a parameter update amplitude.

[0070] To further illustrate the effectiveness and superiority of the efficient hybrid parallel training method for multi-modal models described in the present invention, it is illustrated by the following examples. The multi-modal models used are the CLIP and LIT multi-modal models respectively, and the preprocessed food101 dataset is used. The training set contains 75,750 images and the experiment is carried out on 8 H800 GPUs of a 1-machine 8-card server. Comparing with the existing 1F1B and Gpipe pipeline parallel training methods, the single-round iteration speed is improved, and the relevant data of the model configuration table are shown in Table 1 below.

[0071] Table 1

[0072]

[0073] Combined with Figure 5 and Figure 6 , the comparison graph of training 60 rounds using CLIP with ResNet101 as the image modality and Transformer as the text modality on the Country211 dataset, with the learning rate set to 5e-3, the momentum to 0.9, the weight decay to 1e-4, and the batch size to . Figure 5 is the comparison graph of the iteration time after normalizing the single-iteration time of this method to 1. It can be seen from the graph that the method of the present invention (denoted as HybPipe) has improved compared with the mainstream training strategies, especially more significantly in the environment of two nodes. Taking the LIT model as an example, in the single-node case where Nodes = 1, the single-iteration time of 1F1B and Gpipe is 1.083 times and 1.125 times slower than this method respectively. In the multi-node case where Nodes = 2, the iteration time gap is further enlarged, and the single-iteration time of 1F1F and Gpipe is 1.182 times and 1.257 times slower than the method described in the present invention respectively. In Figure 6In the results, within the same number of iterations, i.e., 60 rounds, the final accuracy rates of 1F1B, asynchronous pipeline Asynchronous, and HybPipe of the present invention are 73.81%, 69.71%, and 74.4%, respectively. It can be seen that the accuracy rate of the model trained using the method of the present invention is consistent with that of the synchronous pipeline strategy, avoiding the problem of accuracy degradation caused by gradient staleness brought about by asynchronous pipeline parallelism.

[0074] The present invention proposes an innovative training method that cleverly combines asynchronous pipeline parallel technology and a phased training strategy. Through this combination, not only the training efficiency is significantly improved, but also the problem of decreased model accuracy that may be caused by asynchronous pipeline parallelism is successfully alleviated. In addition, this method can make full use of hardware resources and provide an efficient and flexible solution for the distributed training of multimodal models. When training a dual-stream multimodal model on two devices with 1 machine and 8 cards, the entire model is distributed to all GPU devices using 16 GPU data parallelism during the warm-up phase; after dividing each modal sub-module of the model into multiple consecutive stages during the acceleration phase, each stage independently occupies a GPU using asynchronous pipeline parallelism. By injecting micro-batches into the pipeline, the overlap of GPU device computing and communication is achieved, the idle time of the GPU device is reduced, and resource utilization is improved.

[0075] The core of the multimodal model is to combine data from different modalities to achieve joint learning and reasoning. By integrating data from multiple modalities into the same model, the model has a more comprehensive perspective when processing complex tasks. However, data from different modalities have significant differences in characteristics, dimensions and structures, and existing distributed model training strategies have many limitations. The present invention provides an efficient hybrid parallel training strategy for multimodal models. By combining data parallelism, asynchronous pipeline parallelism and staged training strategies, it optimizes the heterogeneous characteristics of multimodal models and accelerates the multimodal model training process. The present invention can be applied to the following fields:

[0076] Multimodal model training acceleration: During the training of multimodal models, an efficient hybrid parallel strategy can significantly accelerate the training process, reducing time and resource consumption for large-scale datasets and complex model architectures. This technology is particularly suitable for industrial-grade AI applications that require rapid iteration and deployment.

[0077] Cross-modal data analysis: In scenarios where multiple data types need to be processed simultaneously, the cross-modal reasoning ability of the model is enhanced through efficient parallel training. For example, in audio content analysis, the present invention can simultaneously process text (text modality) and audio tracks (auditory modality), and generate a comprehensive understanding result in combination with the text description. This ability is of great significance for applications such as intelligent monitoring and multimedia content recommendation.

Claims

1. An efficient hybrid parallel training method for multi-modal models, characterized in that, It includes the following steps: Obtain a multi-modal model and the datasets required for training; Conduct an architecture analysis on the multi-modal model. According to the modal types of the input data, divide the multi-modal model into multiple independent modal sub-modules, where each modal sub-module is dedicated to processing a specific modal data; Evaluate the computational load of each modal sub-module when processing global batch size data by statically analyzing the parameters of each modal sub-module; Based on the results of the computational load analysis, formulate a computing power resource allocation plan to determine the number of devices required for each modal sub-module; According to the computing power resource allocation plan, adopt a partitioning algorithm for each modal sub-module to divide the modal sub-module into multiple consecutive pipeline stages; Divide the training process into a warm-up stage and an acceleration stage, and use the datasets to train the multi-modal model by data parallelism and asynchronous pipeline parallelism respectively; The steps of dividing the training process into a warm-up stage and an acceleration stage, and using the datasets to train the multi-modal model by data parallelism and asynchronous pipeline parallelism respectively include: In the first 10% stage of the model training, adopt a data parallel strategy for training, use a linear warm-up learning rate, and when switching stages, the learning rate reaches a preset maximum value. This stage is recorded as the warm-up stage; The remaining training stage is used as the acceleration stage. Switch to the asynchronous pipeline parallel strategy, allocate the multiple pipeline stages divided from the multi-modal model to their respective devices for asynchronous update, and at the same time allocate the training dataset to each modal sub-module; In the acceleration stage, each pipeline stage runs independently on its respective GPU and performs asynchronous forward calculation and backpropagation. At the same time, the learning rate enters the decay stage, and the learning rate shows a smooth decay curve through a cosine function, and finally decays to close to 0, and gradient accumulation is introduced. The calculation results of each pipeline stage are temporarily stored in the local buffer, and when the gradients of a preset number of micro-batches are accumulated on the GPU pipeline, the gradient update is performed uniformly.

2. The efficient hybrid parallel training method for a multi-modal model according to claim 1, wherein The steps of formulating a computing power resource allocation plan based on the results of the computational load analysis to determine the number of devices required for each modal sub-module include: Adopt the proportional allocation principle, and allocate the corresponding number of devices according to the computational load ratio of each modal sub-module and the total number of devices; 3. An efficient hybrid parallel training method for a multimodal model according to claim 1, characterized in that, The steps of adopting a partitioning algorithm for each modal sub-module according to the computing power resource allocation plan to divide the modal sub-module into multiple consecutive pipeline stages include: Take each modal sub-module independently as the input of the partitioning algorithm, analyze the number of parameters of each layer of the modal sub-module, use dynamic programming to determine the optimal partitioning boundary according to the results of the hierarchical analysis, and divide the modal sub-module into multiple consecutive pipeline stages according to the optimal partitioning boundary.

Images