Large and small model cooperative training method and device for multi-modal large language model

Abstract

The invention provides a large and small model cooperative training method and device for a multi-modal large language model. The method comprises the following steps: inputting multi-modal samples such as images, videos and audios into a multi-modal shared encoder to generate a multi-modal token; inputting the text sample into a text encoder to obtain a text token; multi-modal tokens of different modals are combined with text tokens, a pre-trained small multi-modal large language model is used for forward distillation of a pre-trained large multi-modal large language model, a projection layer is trained to obtain a trained projection layer, and the projection layer is connected between the small multi-modal large language model and the large multi-modal large language model; and utilizing the trained projection layer and the pre-trained large-scale multi-modal large language model to reversely distill the small-scale multi-modal large language model, thereby obtaining the trained large-scale and small-scale multi-modal large language models at the same time in the training process. Not only is the cross-modal alignment capability of the large-scale multi-modal large language model improved, but also the performance of the small-scale multi-modal large language model is enhanced through knowledge distillation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for collaborative training of large and small models of a multimodal large language model. Background Art

[0002] In recent years, multimodal, large-scale pre-trained models have achieved significant breakthroughs in the field of artificial intelligence. By integrating visual and linguistic information, these models have demonstrated outstanding performance in complex tasks such as image captioning and visual question answering. However, existing multimodal large-language models (MLLMs) typically follow two independent research paths: one is to improve performance by expanding model size, and the other is to reduce parameters through methods such as pruning to adapt to environments with limited computing resources. The independence of these two paths leads to inefficient training strategies and poor inter-model connectivity, failing to fully leverage the respective strengths of large and small models. Summary of the Invention

[0003] The present invention provides a method and device for collaborative training of large and small models of a multimodal large language model, which is used to address the shortcomings of existing multimodal large language model training strategies, such as low efficiency, poor connectivity between models, and failure to fully utilize the respective advantages of large and small models. The method achieves collaborative training of large multimodal large language models and small multimodal large language models. The technical solutions proposed by the present invention are as follows: In a first aspect, the present invention provides a method for collaboratively training large and small models of a multimodal large language model, comprising: Obtain a multimodal large language model, multimodal samples, and text samples; the multimodal large language model includes a multimodal shared encoder, a text encoder, a pre-trained small multimodal large language model, and a pre-trained large multimodal large language model; the small multimodal large language model and the large multimodal large language model are connected and aligned via a projection layer; the multimodal samples are a combination of images, videos, and audio; Inputting the multimodal samples into the multimodal shared encoder to generate multimodal tokens; Inputting a text sample into the text encoder to obtain a text token; Combining the multimodal token with the text token, forward distilling the pre-trained large multimodal language model through the pre-trained small multimodal language model and training the projection layer to obtain a trained projection layer; The pre-trained small multimodal large language model is reverse-distilled using the trained projection layer and the pre-trained large multimodal large language model to obtain a trained large multimodal large language model and a trained small multimodal large language model.

[0004] Optionally, combining the multimodal tokens with the text tokens, forward distilling the pre-trained large multimodal language model through a pre-trained small multimodal language model, and training the projection layer to obtain the trained projection layer includes: The small multimodal large language model generates corresponding text descriptions based on multimodal tokens and text tokens; The projection layer spatially aligns the multimodal tokens with the text tokens to generate aligned multimodal features; A large multimodal language model generates corresponding text descriptions based on the aligned multimodal features; Calculate the first total loss based on the pre-constructed first loss function and the generated text description; The optimization algorithm is used to update the parameters of the projection layer to minimize the first total loss, and the trained projection layer is obtained.

[0005] Optionally, calculating the first total loss based on the pre-constructed first loss function and the generated text description includes: Calculate the difference between the text description generated by the small multimodal large language model and the true label of the text sample to obtain the first caption loss; Calculate the KL divergence between the probability distribution of the text description generated by the small multimodal large language model and the probability distribution of the text description generated by the large multimodal large language model to obtain the forward KLD loss; The first subtitle loss and the forward KLD loss are input into a pre-constructed first loss function to obtain the first total loss.

[0006] Optionally, the using the trained projection layer and the pre-trained large multimodal language model to reversely distill the pre-trained small multimodal language model to obtain the trained large multimodal language model and the trained small multimodal language model includes: The small multimodal large language model generates corresponding text descriptions based on multimodal tokens and text tokens; Use the trained projection layer to spatially align the multimodal tokens with the text tokens to generate aligned multimodal features; A large multimodal language model generates corresponding text descriptions based on the aligned multimodal features; Calculate the second total loss based on the pre-constructed second loss function and the generated text description; An optimization algorithm is used to update the parameters of the pre-trained small multimodal large language model and the parameters of the pre-trained large multimodal large language model to minimize the second total loss, thereby obtaining a trained large multimodal large language model and a trained small multimodal large language model.

[0007] Optionally, calculating the second total loss based on a pre-constructed second loss function and the generated text description includes: Calculate the difference between the text description generated by the large multimodal language model and the true label of the text sample to obtain the second caption loss; Calculate the KL divergence between the probability distribution of text descriptions generated by the large multimodal large language model and the probability distribution of text descriptions generated by the small multimodal large language model to obtain the inverse KLD loss; The second subtitle loss and the inverse KLD loss are input into a pre-constructed second loss function to obtain the second total loss.

[0008] Optionally, the multimodal token includes a first visual token, a second visual token, and a third visual token, and the multimodal shared encoder includes an image word segmenter, a video word segmenter, an audio word segmenter, and a visual encoder; The step of inputting a multimodal sample into the multimodal shared encoder to generate a multimodal token comprises: The image segmenter segments the image into a plurality of regions to obtain a first image sequence, and the visual encoder encodes the first image sequence to obtain a first visual token; The video segmenter extracts multiple video frames from the video and divides each video frame into multiple regions to obtain a second image sequence. The visual encoder encodes the second image sequence to obtain a corresponding embedded representation, and the embedded representations of multiple video frames are spliced ​​together to obtain a second visual token. The audio word segmenter converts the audio into multiple Mel-spectrograms and divides each Mel-spectrogram into multiple regions to obtain a third image sequence. The visual encoder encodes the third image sequence to obtain an embedded representation, and the embedded representations of multiple Mel-spectrograms are concatenated to obtain a third visual token.

[0009] In a second aspect, the present invention further provides a large-scale model collaborative training device for a multimodal large language model, comprising: An acquisition module is used to acquire a multimodal large language model, multimodal samples, and text samples. The multimodal large language model includes a multimodal shared encoder, a text encoder, a pre-trained small multimodal large language model, and a pre-trained large multimodal large language model. The small multimodal large language model and the large multimodal large language model are connected and aligned via a projection layer. The multimodal samples are a combination of images, videos, and audio. a multimodal encoding module, configured to input a multimodal sample into the multimodal shared encoder to generate a multimodal token; A text encoding module, configured to input a text sample into the text encoder to obtain a text token; a forward distillation module for combining the multimodal tokens with the text tokens, forward distilling the pre-trained large multimodal language model through the pre-trained small multimodal language model and training the projection layer to obtain a trained projection layer; The reverse distillation module is used to reversely distill the pre-trained small multimodal large language model using the trained projection layer and the pre-trained large multimodal large language model to obtain the trained large multimodal large language model and the trained small multimodal large language model.

[0010] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for collaborative training of large and small models of a multimodal large language model as described in the first aspect above is implemented.

[0011] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the large and small model collaborative training method of the multimodal large language model as described in the first aspect above.

[0012] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the large and small model collaborative training method of the multimodal large language model as described in the first aspect above.

[0013] Based on the above technical solution, the present invention has the following beneficial effects compared with the prior art: The present invention provides a method and device for collaborative training of large and small models of a multimodal large language model. Through a collaborative training framework, a large multimodal large language model and a small multimodal large language model are connected through a projection layer. The lightweight characteristics of the small multimodal large language model can be used to quickly align multimodal information, thereby improving the overall training efficiency. Then, knowledge distillation is used to help the large multimodal large language model better align cross-modal information, and the rich knowledge base of the large multimodal large language model is used to enhance the performance of the small multimodal large language model. The collaborative training of the large multimodal large language model and the small multimodal large language model not only improves the cross-modal alignment capability of the large multimodal large language model, but also enhances the performance of the small multimodal large language model through knowledge distillation, thereby achieving a close combination of the large multimodal large language model and the small multimodal large language model, and enhancing each other's capabilities, allowing direct adaptation to various computing resource scenarios.

[0014] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0015] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a flowchart of the large and small model collaborative training method of the multimodal large language model provided by the present invention.

[0018] Figure 2 This is a principle framework diagram of the collaborative training of large and small multimodal language models provided by the present invention.

[0019] Figure 3 This is a schematic diagram of the architecture collaboration of the large and small multimodal language models provided by the present invention.

[0020] Figure 4 This is a schematic diagram of the collaborative training strategy provided by the present invention.

[0021] Figure 5 It is a structural diagram of the large and small model collaborative training device for the multimodal large language model provided by the present invention.

[0022] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0024] Existing research on multimodal large language models (MLLMs) is primarily divided into two independent paths: one involves increasing model size to improve performance, and the other focuses on reducing parameters through pruning and other methods to adapt to environments with limited computing resources. Both large and small models have their own advantages, and collaborative training can produce better results compared to independent training strategies.

[0025] The mainstream approach in MLLM research is to increase the model size and provide more data to achieve better scalability. As the model size increases, the demand for computing resources also increases, making MLLM inference and deployment less efficient and more challenging. Therefore, recent research has focused on improving the inference efficiency and deployment ease of MLLM, including methods such as model compression, pruning, and distillation to address these challenges.

[0026] However, these two mainstream research directions are independent and lack integration. The current training strategy is relatively inefficient, the connectivity between models is poor, and the respective advantages of large multimodal large language models and small multimodal large language models are not effectively utilized. In other words, there is a lack of a comprehensive method to utilize the advantages of both to mutually enhance the training process. To this end, the present invention proposes a new collaborative training strategy for large models and small models, which combines these two training methods to develop a more effective training framework to simultaneously improve the performance of large models and small models. In the collaborative training framework of the present invention, the large multimodal large language model can impart better knowledge to the small multimodal large language model through knowledge distillation, while the small multimodal large language model can assist the large multimodal large language model through multimodal alignment and providing a wider range of multimodal information feature representations than a single model setting.

[0027] The following combination Figures 1-6 The present invention describes a method and device for collaborative training of large and small models of a multimodal large language model.

[0028] This paper adopts a novel collaborative training framework, connecting a large multimodal language model and a small multimodal language model through a projection layer. The lightweight nature of the small multimodal language model is leveraged to effectively align multimodal information. Knowledge distillation is then used to help the large multimodal language model better align cross-modal information. During the fine-tuning phase, the rich knowledge base of the large multimodal language model is leveraged to enhance the performance of the small multimodal language model.

[0029] The training process of a multimodal model consists of two phases: alignment and fine-tuning. These phases achieve collaboration between the small multimodal language model and the large multimodal language model. In the alignment phase, the small multimodal language model pre-aligns multimodal information, specifically aligning multimodal image features with text features across images, videos, and audio. This helps the large multimodal language model achieve better image-text alignment. In the fine-tuning phase, the large multimodal language model leverages its richer knowledge to perform knowledge distillation using reverse Kullback-Leibler divergence (KLD) to guide learning in the small multimodal language model.

[0030] Figure 1 A flow chart of a method for collaboratively training large and small models of a multimodal large language model provided by the present invention, the method comprising the following steps: Step S101: Obtain a large multimodal language model, multimodal samples, and text samples.

[0031] The multimodal large language model includes a multimodal shared encoder, a text encoder, a pre-trained small multimodal large language model, and a pre-trained large multimodal large language model. The multimodal samples are a combination of images, videos, and audio.

[0032] Reference Figure 2 As shown, the small multimodal large language model described above has a relatively small parameter size, making it suitable for resource-constrained environments or rapid deployment scenarios. The pre-training method for the small multimodal large language model involves collecting a large amount of text data as training corpus. Pre-training is performed using unsupervised or weakly supervised learning tasks. Efficient optimization algorithms (such as Adam) and pre-training strategies (such as warm-start and progressive training) are employed to accelerate the training process.

[0033] These large multimodal language models have a large number of parameters, enabling them to handle more complex language tasks and exhibiting stronger generalization capabilities. The pre-training method for large multimodal language models is similar to that for small multimodal language models, but requires larger amounts of training data. Pre-training also utilizes unsupervised or weakly supervised learning tasks, employing efficient optimization algorithms (such as Adam) and pre-training strategies (such as warm-start and progressive training) to accelerate the training process.

[0034] The small multimodal language model and the large multimodal language model are connected and aligned via a projection layer, transforming and aligning the feature space. This facilitates information transfer and adaptation between language models of different sizes. Linear projection, multi-layer perceptron (MLP), or cross-attention mechanisms can be used as the projection layer.

[0035] Step S102: Input the multimodal sample into the multimodal shared encoder to generate a multimodal token.

[0036] Reference Figure 3 As shown, multimodal samples containing images, videos, and audio are input into the multimodal shared encoder. The multimodal shared encoder processes these input data and generates image tokens representing different modalities (images, video frames, audio features, etc.).

[0037] Step S103: Input the text sample into the text encoder to obtain a text token.

[0038] Reference Figure 3As shown, a text sample is input into a text encoder, which converts the text into a series of text tokens that capture the semantic and syntactic information in the text.

[0039] Step S104: Combine the multimodal token with the text token, forward distill the pre-trained large multimodal language model through the pre-trained small multimodal language model, and train the projection layer to obtain a trained projection layer.

[0040] In this phase, also known as the alignment phase, a small multimodal language model is used to align multimodal image-text features across images, videos, and audio. Generative learning is used to rapidly align visual and language features. Knowledge distillation techniques are then used to assist the large multimodal language model in cross-modal alignment. During this process, the small multimodal language model uses a forward KLD loss to guide the learning of the large multimodal language model. Training the projection layer helps the large multimodal language model better align multimodal information. Specifically, the image tokens generated by the multimodal shared encoder are combined with the text tokens generated by the text encoder to form a multimodal input representation. This multimodal input representation is forward propagated through the pre-trained small multimodal language model and simultaneously distilled to the pre-trained large multimodal language model. During this process, the projection layer is trained to optimize the large multimodal language model's ability to understand and process multimodal information. The projection layer adjusts the feature space output by the small multimodal language model to better match the input requirements of the large multimodal language model.

[0041] Step S105: using the trained projection layer and the pre-trained large multimodal language model, reverse distilling the pre-trained small multimodal language model to obtain a trained large multimodal language model and a trained small multimodal language model.

[0042] In this stage, also known as the fine-tuning stage mentioned above, the knowledge base of the large multimodal language model is used to guide the learning of the small multimodal language model through the reverse KLD loss, thereby enhancing the multimodal understanding ability of the small multimodal language model and making the small multimodal language model perform better. Specifically, the knowledge in the large multimodal language model is reverse distilled back to the small multimodal language model using the trained projection layer and the pre-trained large multimodal language model. Through this process, the small multimodal language model is able to absorb the advantages of the large multimodal language model in processing multimodal information while maintaining its original lightness and efficiency. At the same time, the large multimodal language model further optimizes its cross-modal processing capabilities through synergy with the projection layer.

[0043] The modal model collaborative training method provided by the present invention connects the large multimodal large language model and the small multimodal large language model through a projection layer through a collaborative training framework. The lightweight characteristics of the small multimodal large language model can be used to quickly align multimodal information, thereby improving the overall training efficiency. Then, knowledge distillation is used to help the large multimodal large language model better align cross-modal information, and the rich knowledge base of the large multimodal large language model is used to enhance the performance of the small multimodal large language model. The collaborative training of the large multimodal large language model and the small multimodal large language model not only improves the cross-modal alignment capability of the large multimodal large language model, but also enhances the performance of the small multimodal large language model through knowledge distillation, so that it can perform well in an environment with limited computing resources. The effective processing of multimodal information and the transfer and complementation of cross-modal knowledge are achieved, which is of great significance for improving model performance and generalization ability. The collaborative training method of the present invention enables a single training session to simultaneously generate a large multimodal language model and a small multimodal language model. It can directly adapt to various computing resource scenarios (large computing resource cloud scenarios and small computing resource end-side scenarios) without the need for successive fine-tuning training and compression distillation, and has broad application prospects.

[0044] Moreover, through the collaborative training of a multimodal shared encoder, a text encoder, and language models of different sizes, the model can better understand and process information from different modalities, thereby improving overall performance. The forward distillation and backward distillation processes achieve the transfer and complementarity of knowledge between different models, allowing the small multimodal large language model to absorb the advantages of the large multimodal large language model, while the large multimodal large language model also further optimizes its cross-modal processing capabilities through adjustments to the projection layer. The trained small multimodal large language model maintains its original lightness and efficiency, making it easy to deploy and use in practical applications. At the same time, the large multimodal large language model also improves its flexibility and accuracy in cross-modal processing through collaborative training. In addition, the introduction of multimodal information enables the model to handle more complex and diverse input data, thereby enhancing its generalization ability in practical applications.

[0045] This paper uses CLIP-VIT-L / 14 as a multimodal shared encoder and the lightweight language model OPT-125m as a small multimodal language model. Furthermore, Vicuna-7B is selected as the large multimodal language model. The large multimodal language model and the small multimodal language model are connected via projection layers (e.g., two linear layers) to achieve cross-modal feature alignment. In the collaborative training framework, CLIP-VIT-L / 14, as the multimodal shared encoder, processes the input multimodal images (images, videos, and audio), OPT-125m, as the small multimodal language model, processes the input text, and Vicuna-7B, as the large multimodal language model, performs the final cross-modal task processing. The collaborative training method proposed in this paper rapidly aligns the multimodal shared encoder and the small multimodal language model. Subsequently, a distillation strategy is employed to help the large multimodal language model align multimodal information. During the fine-tuning phase, the multimodal features output by the small multimodal language model enhance the multimodal understanding capabilities of the large multimodal language model. In addition, the large multimodal big language model can impart its rich knowledge base to the small multimodal big language model.

[0046] In an alternative embodiment, referring to Figure 3 As shown, a single encoder is shared across image, video, and audio modalities to achieve better multimodal unified representation and more comprehensive modal alignment. The backbone of this multimodal shared encoder network is a pre-trained visual encoder, fully shared by the image, video, and audio modalities. These modalities differ in their tokenizers, hereinafter referred to as the image tokenizer, video tokenizer, and audio tokenizer, respectively. The tokenizer divides the input into a sequence of tokens for the visual encoder input. Correspondingly, the image tokens corresponding to the image, video, and audio are referred to as the first visual token, the second visual token, and the third visual token, respectively.

[0047] The step S102 of inputting the multimodal sample into the multimodal shared encoder to generate a multimodal token includes: S1021. The image segmenter divides the image into multiple regions to obtain a first image sequence, and the visual encoder encodes the first image sequence to obtain a first visual token.

[0048] First, the image tokenizer segments the input image into multiple regions. This step divides the image into a fixed-size grid, where each grid region is considered a candidate for a feature unit. These segmented image regions (i.e., the first image sequence) are then fed into a visual encoder for encoding. A visual encoder, such as the Vision Transformer (ViT) or other convolutional neural network (CNN)-based encoders, extracts features from each image region and converts these features into a high-dimensional embedding representation (i.e., the first visual token mentioned above).

[0049] S1022. The video segmenter extracts multiple video frames from the video and divides each video frame into multiple regions to obtain a second image sequence. The visual encoder encodes the second image sequence to obtain a corresponding embedded representation, and the embedded representations of the multiple video frames are spliced ​​to obtain a second visual token.

[0050] The video tokenizer extracts multiple keyframes or all frames from the video stream. For each video frame, the tokenizer performs the same segmentation operation as image processing, dividing the video frame into multiple regions to form a second image sequence. The visual encoder encodes each video frame in the second image sequence separately, generating an embedded representation for each video frame. These embedded representations are then concatenated in chronological order to form a second visual token that contains the overall video information. This concatenation preserves the temporal dimension of the video, helping the model understand dynamic changes in the video.

[0051] S1023. The audio word segmenter converts the audio into multiple mel-spectrograms, and divides each mel-spectrogram into multiple regions to obtain a third image sequence. The visual encoder encodes the third image sequence to obtain an embedded representation, and concatenates the embedded representations of the multiple mel-spectrograms to obtain a third visual token.

[0052] The audio tokenizer converts the audio signal into a series of mel-spectrograms. A mel-spectrogram is a two-dimensional representation of audio that captures both frequency and temporal information. The tokenizer then segments each mel-spectrogram into multiple regions, similar to how images and videos are processed, to form a third image sequence. The visual encoder also encodes each mel-spectrogram in the third image sequence, generating an embedded representation for each spectrogram. These spectrogram embeddings are then concatenated in a specific order (e.g., chronological order or the order in which the spectrograms were generated) to form a third visual token that contains the overall audio information.

[0053] By converting image, video and audio data into similar image token representations, the present invention enables the multimodal shared encoder to more easily achieve cross-modal fusion in subsequent processing stages. This unified representation reduces the complexity of the model in processing multimodal data. Due to the use of configurable word segmenters and visual encoders, the processing flow can flexibly adapt to image, video and audio inputs of different sizes and resolutions. In video processing, by retaining the embedded representation of video frames and splicing them in chronological order, the model can capture the temporal dimension information in the video. Moreover, the audio signal is innovatively converted into a Mel-spectrogram and processed in a similar image manner, providing a novel and effective representation method for audio data. This representation method not only retains the spectral information of the audio, but also further extracts high-level features through the processing of the visual encoder.

[0054] In addition, since the multimodal shared encoder can process multiple types of media data and generate a unified token representation, it can be easily combined with the text encoder to build a more powerful multimodal model.

[0055] Unlike traditional vision-only backbone and dual-encoder models, the collaborative framework of our invention comprises a unified multimodal shared encoder and two tightly integrated language models of different sizes. The large multimodal LLM and the small multimodal LLM are essentially mirror images of each other, differing primarily in the number of parameters. This symmetry is advantageous because it allows for efficient training of both models using a single infrastructure and optimization techniques. We use two projection layers to map modal features into the semantic space of the LLM. For the large multimodal LLM, we selected the Vicuna-7B, while for the small multimodal LLM, we selected the OPT-125m. This configuration is crucial for validating the effectiveness of our collaborative training approach, which aims to harmonize the small and large multimodal LLMs. To bridge the significant modality and structural differences between the two language models and thereby mutually enhance their performance, we employ a progressive collaborative training strategy. This strategy begins with pre-training the small multimodal LLM, then generatively learns to synchronize other visual and language modalities, such as image, video, and audio, ultimately transitioning to full collaborative training using knowledge distillation. The collaborative training strategy of the present invention consists of two phases: feature alignment and instruction fine-tuning. In the feature alignment phase, the small multimodal large language model rapidly aligns visual and linguistic features of other modalities, such as images, video, and audio, by leveraging its smaller parameter size. Subsequently, the present invention employs distillation techniques to assist the large multimodal large language model in achieving alignment. In the fine-tuning phase, the large multimodal large language model leverages its powerful linguistic capabilities to enrich the knowledge of the small multimodal large language model through a reverse distillation method.

[0056] This paper uses generative learning to align OPT125m (a small multimodal large language model) and CLIP-vit (a multimodal shared encoder), focusing on the embedding of multimodal image-text pairs across images, videos, and audio. Because their parameter sizes are comparable, small multimodal large language models can be used more efficiently to align image, video, and audio-image multimodal text pairs. Furthermore, because they share the same token embeddings, distillation can be used to better facilitate cross-modal alignment of large multimodal large language models. The existing LAION-400M dataset provides 80 million cleaned data samples. During training, text samples are encoded into Tf using OPT125m, and multimodal samples are extracted into If using CLIP-vit.

[0057] Based on the objective function of BLIP-2, the present invention uses multiple loss terms when training a multimodal pre-trained model. These include the ITC loss for learning the correspondence between images, video, audio, and text; the ITM loss for determining whether images and text match; and the ITG loss for optimizing the model's ability to generate text from multimedia content. These loss terms work together to enhance the model's multimodal understanding and generation capabilities. This process allows for the extraction of powerful multimodal representations, more closely aligning the feature space with the visual encoder. Unlike QFormer, the small multimodal large language model inherently possesses multimodal understanding and reasoning capabilities. It can not only generate subtitles but also perform various QA tasks. Text tokens are generated sequentially, guiding the large multimodal large language model and the small multimodal large language model. The small multimodal large language model achieves strong multimodal capabilities through its initial alignment with the multimodal shared encoder. While the large multimodal large language model possesses powerful language capabilities, its limited cross-modal alignment capabilities cause it to lag behind the small multimodal large language model in multimodal understanding and reasoning. Therefore, we chose to use a small multimodal language model to guide the large multimodal language model. By leveraging the cross-modal alignment of the small multimodal language model in the first stage, we aim to enhance the image, video, and audio multimodal image-text understanding capabilities of the large multimodal language model.

[0058] In the collaborative training framework of the present invention, distillation and reverse distillation techniques are used to connect large multimodal language models and small multimodal language models, so that they can learn from each other's advantages and thus improve each other's performance. The present invention considers conditional text generation, and the model is based on the probability distribution. 、 Sampling 、 Generate responses, which is usually how large multimodal language models perform the task. We formalize knowledge distillation (KD) as an optimization problem that aims to minimize the fixed teacher model distribution and by parameters Parameterized Student model distribution The standard KD method minimizes the forward KLD:

[0059] KL divergence KL[p‖ ]:express and The smaller its value, the closer the two distributions are.

[0060] : Indicates that under the real distribution, given hour The conditional probability of .

[0061] : represents the distribution of the parameterized Student model Next, given hour The conditional probability of .

[0062] However, knowledge distillation (KD) is particularly suitable for large multimodal language models because they perform generative tasks where low-capacity student models struggle to replicate the complex text generation patterns of teacher models or humans. To alleviate these challenges for the student model, the present invention uses reverse KLD in the second stage. The learning objective of the large multimodal language model involves minimizing the reverse KLD between the student model and the teacher model distribution:

[0063] Minimizing the inverse KLD is thought to induce pattern-seeking behavior in generative modeling, where the model Assign high probability to The significant mode of , while ignoring the smaller modes. This phenomenon has been documented in some studies. In contrast, minimizing the forward KLD leads to Assigning a large probability mass to The zero probability area of ​​​​, thus generating low-quality text in practical applications. However, reverse KLD gives priority to The main mode of the network plays an important role in ensuring the accuracy and reliability of text generation. Unlike the sequence-level knowledge distillation that minimizes the forward KLD, minimizing the reverse KLD does not force Meet the distribution from the faculty All samples in Instead, it forces the student model to generate samples that match the teacher's preferences within its capabilities, which is easier to achieve for a small multimodal large language model.

[0064] The above step S104 combines the multimodal token with the text token, referring to Figure 4 As shown, the pre-trained large multimodal language model is forward-distilled through the pre-trained small multimodal language model and the projection layer is trained to obtain the trained projection layer, which specifically includes: S1041. The small multimodal large language model generates corresponding text descriptions based on the multimodal tokens and text tokens.

[0065] The input to the small multimodal large language model includes multimodal tokens (i.e., the first visual token, the second visual token, and the third visual token) and text tokens. The small multimodal large language model receives these multimodal image tokens and text tokens as input and generates corresponding text descriptions based on these inputs. The small multimodal large language model is typically a model with fewer parameters and higher computational efficiency. Using its own parameters and pre-trained knowledge, the small multimodal large language model fuses and processes these multimodal tokens to generate a preliminary text output. The small multimodal large language model outputs the generated text description, which will be used for loss calculation and model optimization in subsequent steps.

[0066] S1042. The projection layer spatially aligns the multimodal tokens with the text tokens to generate aligned multimodal features.

[0067] The projection layer receives multimodal tokens and text tokens and spatially aligns these tokens from different modalities to generate a unified feature representation rich in multimodal information. This alignment process involves multiple techniques, such as feature transformation, attention mechanisms, and fusion strategies, to ensure that features from different modalities can be compared and combined in the same space. The projection layer outputs aligned multimodal features, which serve as input to the large multimodal language model. The alignment process can be found in the description of the projection layer in the prior art and will not be repeated here.

[0068] S1043. The large multimodal language model generates corresponding text descriptions based on the aligned multimodal features.

[0069] The large multimodal language model receives the aligned multimodal features from the projection layer as input. Leveraging its powerful parameters and pre-trained knowledge, the large multimodal language model deeply processes and analyzes these multimodal features, generating a more accurate, rich, and fluent text description. Because large multimodal language models typically have more parameters and greater expressive power, they are able to capture more detail and semantic information. The large multimodal language model outputs the generated text description, which is used together with the text description generated by the small multimodal language model for loss calculation and model optimization in subsequent steps.

[0070] S1044. Calculate a first total loss based on a pre-constructed first loss function and the generated text description; This step requires the text descriptions generated by the small multimodal large language model and the large multimodal large language model, as well as the true labels of the text samples. First, the difference between the text descriptions generated by the small multimodal large language model and the true labels is calculated (e.g., using cross-entropy loss) to obtain the first caption loss. Next, the KL divergence (forward KLD loss) between the probability distribution of the text descriptions generated by the small multimodal large language model and the probability distribution of the text descriptions generated by the large multimodal large language model is calculated to measure the difference between the two. The first caption loss measures the direct difference between the model-generated text and the true text, while the forward KLD loss encourages the text distribution generated by the small multimodal large language model to converge with the distribution of the large multimodal large language model, achieving knowledge distillation. Finally, these two loss terms are weighted and summed according to predetermined weights to obtain the first total loss. The calculated first total loss value is output and will be used for model optimization in subsequent steps.

[0071] S1045. Use an optimization algorithm to update the parameters of the projection layer to minimize the first total loss, and obtain a trained projection layer.

[0072] The projection layer parameters are updated using an optimization algorithm (such as gradient descent or Adam) to minimize the first total loss. This update process is repeated multiple times until the loss converges or the preset number of training rounds is reached. The trained projection layer parameters are output, which enable the projection layer to better align multimodal tokens with text tokens, thereby improving the performance of the entire multimodal model. This process results in an optimized projection layer that can more effectively spatially align multimodal tokens with text tokens and generate feature representations rich in multimodal information, thereby improving the text generation and comprehension capabilities of the multimodal model.

[0073] Through the spatial alignment of the projection layers, the present invention enables the model to better understand the relationship between different modalities, thereby improving the understanding and processing capabilities of multimodal data. By forward distilling a large multimodal language model using a small multimodal language model, the complexity and computational cost of the model can be reduced while maintaining a certain performance, thereby achieving model compression and acceleration. With the participation of a large multimodal language model, the model can generate more accurate, rich and fluent text descriptions, improving the quality and diversity of text generation. The model of the present invention can process and understand multimedia data from different sources, including static images, dynamic videos, and audio information. This multimodal processing capability enables the model to more comprehensively understand complex scenes in the real world. This method is not only suitable for cross-modal processing of images and text, but can also be extended to other modalities such as video and audio, providing technical support for applications such as cross-modal retrieval, generation, and understanding.

[0074] In an optional embodiment, the calculation of the first total loss based on the pre-constructed first loss function and the generated text description in step S1044 specifically includes: S201. Calculate the difference between the text description generated by the small multimodal large language model and the true label of the text sample to obtain the first subtitle loss.

[0075] First, a small multimodal large language model generates a text description based on the input image, video or audio multimodal image token and text token. Then, the generated text description is compared with the true label of the text sample (i.e. Figure 4 The first caption loss is calculated using a difference metric such as cross-entropy loss. This loss measures the degree of difference between the generated text and the real text.

[0076] S202. Calculate the KL divergence between the probability distribution of the text description generated by the small multimodal large language model and the probability distribution of the text description generated by the large multimodal large language model to obtain the forward KLD loss.

[0077] The large multimodal language model generates a text description based on the multimodal features aligned by the projection layer. The KL divergence (Kullback-Leibler Divergence) is calculated between the probability distribution of the text description generated by the small multimodal language model and the probability distribution of the text description generated by the large multimodal language model. KL divergence is a method for measuring the difference between two probability distributions, asymmetrically measuring the information loss of one distribution relative to the other. The forward KLD loss encourages the text distribution generated by the small multimodal language model to converge with the distribution of the large multimodal language model, achieving knowledge distillation.

[0078] S203: Input the first subtitle loss and the forward KLD loss into a pre-constructed first loss function to obtain the first total loss.

[0079] During the training phase, as shown in Equation 1, forward distillation facilitates the guidance of the representation learning of the large multimodal language model by the small multimodal language model. Probabilistically guided supervision enables the large multimodal language model to effectively absorb the multimodal understanding capabilities of the small multimodal language model. Simultaneously, the parameters of the large multimodal language model are frozen to prevent catastrophic forgetting. During this phase, only the projection layers inserted into the large multimodal language model are trained. The total loss at this stage consists of the first caption loss and the forward KLD loss. The first caption loss and the forward KLD loss are weighted and summed according to the loss function in Equation 1 to obtain the first total loss.

[0080]

[0081] Among them, λ is a balance coefficient, which is 0.3 in this stage.

[0082] The present invention directly optimizes the similarity between the text generated by a small multimodal large language model and the true label through the first subtitle loss, which helps to improve the accuracy and naturalness of text generation. The forward KLD loss achieves knowledge distillation by encouraging the small multimodal large language model to imitate the output distribution of the large multimodal large language model. This enables the small multimodal large language model to reduce computational complexity and resource consumption while maintaining a certain level of performance. Because the loss function is constructed based on multimodal input (image, video, audio) and text output, the training process enhances the model's understanding and processing capabilities of multimodal data. By combining different loss terms, namely the subtitle loss and the KLD loss, the model's trade-off between generation accuracy and distillation effect can be flexibly adjusted to adapt to different application scenarios and needs. This training strategy is not only applicable to current multimodal models, but can also be extended to other types of multimodal tasks or models, facilitating model expansion and migration.

[0083] In an optional embodiment, the above step S105 utilizes the trained projection layer and the pre-trained large multimodal language model to reversely distill the pre-trained small multimodal language model (refer to Figure 4 As shown), the trained large multimodal language model and the trained small multimodal language model are obtained, including: S1051. The small multimodal large language model generates corresponding text descriptions based on the multimodal tokens and text tokens.

[0084] The small multimodal large language model receives image tokens and text tokens from different modalities (images, video frames, audio) as input. Based on these inputs, the small multimodal large language model generates corresponding text descriptions, which is a preliminary evaluation of the current capabilities of the small multimodal large language model.

[0085] S1052. Use the trained projection layer to spatially align the multimodal tokens with the text tokens to generate aligned multimodal features.

[0086] The trained projection layer is used to spatially align the multimodal tokens with the text tokens to generate aligned multimodal features. These features fuse information from different modalities and are in the same space for easy subsequent processing.

[0087] S1053. The large multimodal language model generates corresponding text descriptions based on the aligned multimodal features.

[0088] The aligned multimodal features are fed into a large multimodal language model. Leveraging its powerful parameters and pre-trained knowledge, the large multimodal language model deeply processes these multimodal features and generates corresponding text descriptions. These text descriptions are typically more accurate, richer, and more fluent than those generated by a smaller multimodal language model.

[0089] S1054. Calculate a second total loss based on a pre-constructed second loss function and the generated text description; This step requires the text descriptions generated by the small multimodal language model and the large multimodal language model, as well as the true labels of the text samples. First, the difference between the text descriptions generated by the large multimodal language model and the true labels is calculated (e.g., using a cross-entropy loss) to obtain the second caption loss. Then, the KL divergence (inverse KLD loss) between the probability distribution of the text descriptions generated by the large multimodal language model and the probability distribution of the text descriptions generated by the small multimodal language model is calculated to measure the difference between the two. The first term, also known as the second caption loss, measures the difference between the text descriptions generated by the large multimodal language model and the true labels of the text samples. This helps maintain the large multimodal language model's ability to fit the real text. The second term, also known as the inverse KLD loss, measures the KL divergence between the probability distribution of the text descriptions generated by the large multimodal language model and the probability distribution of the text descriptions generated by the small multimodal language model. Unlike S104 above, this term encourages the small multimodal language model to learn from the large multimodal language model and optimize its output distribution. Finally, these two loss terms are weighted and summed according to predetermined weights to obtain the first total loss. Output the calculated first total loss value, which will be used for model optimization in subsequent steps.

[0090] S1055. Use an optimization algorithm to update the parameters of the pre-trained small multimodal large language model and the parameters of the pre-trained large multimodal large language model to minimize the second total loss, and obtain the trained large multimodal large language model and the trained small multimodal large language model.

[0091] Use an optimization algorithm (such as gradient descent, Adam, etc.) to update the parameters of the pre-trained small multimodal large language model and (optionally) the large multimodal large language model to minimize the second total loss. This process is repeated multiple times until the second total loss converges or the preset training epochs are reached.

[0092] At this stage, the present invention uses the reverse KLD method to distill the small multimodal large language model through the large multimodal large language model as the teacher model. Since both the small multimodal large language model and the large multimodal large language model have acquired strong multimodal understanding capabilities during the second stage of training, the rich knowledge base of the large multimodal large language model becomes indispensable in this stage. During the fine-tuning process, the large multimodal large language model uses the reverse KLD loss to guide the small multimodal large language model to obtain more powerful multimodal capabilities. Except for the multimodal shared encoder, all parameters, namely the parameters of the large and small multimodal large language models and the text encoder, are trainable.

[0093] By reverse distilling a large multimodal language model, the present invention enables a small multimodal language model to learn more advanced language representation and generation capabilities, thereby improving the accuracy and fluency of its text generation. This is particularly important in resource-constrained environments or scenarios requiring fast reasoning. Because the entire training process is based on multimodal input, the optimized model (whether a small multimodal language model or a large multimodal language model) has stronger multimodal understanding capabilities and can better process and understand data from different modalities.

[0094] In an optional embodiment, the calculation of the second total loss based on the pre-constructed second loss function and the generated text description in step S1054 specifically includes: S301. Calculate the difference between the text description generated by the large multimodal language model and the true label of the text sample to obtain a second subtitle loss.

[0095] The large multimodal language model receives the aligned multimodal features as input and generates corresponding text descriptions. The text descriptions generated by the large multimodal language model are compared with the ground-truth labels of the text samples. A difference metric, such as the cross-entropy loss, is used to calculate the difference between the two, resulting in a second caption loss. This loss measures the similarity or difference between the text generated by the large multimodal language model and the ground-truth text, helping to maintain the large multimodal language model's ability to fit the ground-truth text.

[0096] S302. Calculate the KL divergence between the probability distribution of the text description generated by the large multimodal large language model and the probability distribution of the text description generated by the small multimodal large language model to obtain the reverse KLD loss.

[0097] The KL divergence between the probability distribution of the text descriptions generated by the large multimodal language model and the probability distribution of the text descriptions generated by the small multimodal language model is calculated, but the direction is from the small multimodal language model to the large multimodal language model, hence the name "inverse KLD loss". This loss encourages the output distribution of the small multimodal language model to converge to the output distribution of the large multimodal language model, thereby achieving effective knowledge distillation.

[0098] S303: Input the second subtitle loss and the reverse KLD loss into a pre-constructed second loss function to obtain the second total loss.

[0099] A second loss function is predefined. This function takes the second subtitle loss and the reverse KLD (reKLD) loss as input and combines them into a single loss value, the second total loss, according to certain rules, such as weighted summation. The second subtitle loss and the reverse KLD loss are input into the pre-built second loss function, and the second total loss is calculated according to the function definition.

[0100]

[0101] Among them, λ is a balance coefficient, which is 0.3 in this stage.

[0102] By adjusting the weights of each loss in the second loss function, the optimization degree of the large multimodal large language model and the small multimodal large language model can be balanced according to the requirements of the specific task, thereby obtaining the best model performance.

[0103] The aforementioned inverse KLD loss encourages the small multimodal language model to learn from the output distribution of the large multimodal language model, thereby absorbing its advanced language representation and generative capabilities. This helps improve the accuracy and fluency of the small multimodal language model in text generation tasks. The second subtitle loss ensures that the large multimodal language model does not lose its ability to fit real text during the distillation process. By optimizing this loss, the large multimodal language model can continue to maintain its strong generative capabilities. The entire process achieves knowledge transfer from the large multimodal language model to the small multimodal language model, enabling the small multimodal language model to achieve performance close to that of the large multimodal language model while maintaining low computational complexity. Because the distillation process is based on multimodal input, the optimized model may have stronger multimodal understanding capabilities and be better able to process and understand data from different modalities. Moreover, by optimizing both models simultaneously, training data can be more efficiently utilized, training time can be reduced, and the model's generalization ability can be improved.

[0104] Through the collaborative training strategy of the present invention, the large multimodal language model benefits from the more detailed and informative multimodal semantic features generated by the small multimodal language model, while the small multimodal language model benefits from the knowledge guidance of the large multimodal language model, thereby helping each other improve performance. In addition, the small multimodal language model of the present invention can be directly deployed for inference without the need for new fine-tuning training for each task.

[0105] This paper addresses the shortcomings of existing independent training strategies by proposing a novel collaborative framework that enables collaborative training of a micro (i.e., a small multimodal language model) and a large multimodal language model (i.e., a large multimodal language model). By connecting the large and small multimodal language models via a projection layer, this collaborative training strategy leverages the strengths of each model and improves their performance. Compared to previous approaches, this framework offers several advantages: it strengthens the connection between the large and micro models, improving training efficiency. In the collaborative training strategy, image-text features are effectively aligned by leveraging the parameter efficiency of the small multimodal language model. Knowledge distillation is used to help the large multimodal language model better align cross-modal information. During the fine-tuning phase, the extensive knowledge base of the large multimodal language model is leveraged to enhance the performance of the small multimodal language model. This collaborative training approach achieves a close integration of the large language model and the small multimodal language model, mutually reinforcing their capabilities and allowing for straightforward adaptation to various computing resource scenarios.

[0106] The following describes the large and small model collaborative training device for a multimodal large language model provided by the present invention. The large and small model collaborative training device for a multimodal large language model described below and the large and small model collaborative training method for a multimodal large language model described above can refer to each other.

[0107] Reference Figure 5 As shown, the large and small model collaborative training device of the multimodal large language model provided by the present invention includes: Acquisition module 401 is used to acquire multimodal samples and text samples from a multimodal large language model; wherein the multimodal large language model includes a multimodal shared encoder, a text encoder, a pre-trained small multimodal large language model, and a pre-trained large multimodal large language model, the small multimodal large language model and the large multimodal large language model are connected and aligned via a projection layer, and the multimodal samples are a combination of images, videos, and audio; A multimodal encoding module 402, configured to input a multimodal sample into the multimodal shared encoder to generate a multimodal token; A text encoding module 403 is used to input a text sample into the text encoder to obtain a text token; a forward distillation module 404 for combining the multimodal tokens with the text tokens, forward distilling the pre-trained large multimodal language model through the pre-trained small multimodal language model, and training the projection layer to obtain a trained projection layer; The reverse distillation module 405 is used to reversely distill the pre-trained small multimodal large language model using the trained projection layer and the pre-trained large multimodal large language model to obtain the trained large multimodal large language model and the trained small multimodal large language model.

[0108] Figure 6 An example of a physical structure diagram of an electronic device is shown below. Figure 6 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the large and small model collaborative training method for the multimodal large language model.

[0109] Furthermore, the logic instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0110] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the large and small model collaborative training method of the multimodal large language model provided by the above methods.

[0111] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the large and small model collaborative training method of the multimodal large language model provided by the above methods.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0113] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for collaborative training of large and small models of a multimodal large language model, characterized in that: include: Obtain a multimodal large language model, multimodal samples and text samples; wherein the multimodal large language model includes a multimodal shared encoder, a text encoder, a pre-trained small multimodal large language model and a pre-trained large multimodal large language model, the small multimodal large language model and the large multimodal large language model are connected and aligned through a projection layer, and the multimodal sample is a combination of images, videos and audios; Inputting the multimodal samples into the multimodal shared encoder to generate multimodal tokens; Inputting a text sample into the text encoder to obtain a text token; Combining the multimodal token with the text token, forward distilling the pre-trained large multimodal language model through the pre-trained small multimodal language model and training the projection layer to obtain a trained projection layer; The pre-trained small multimodal large language model is reversely distilled by using the trained projection layer and the pre-trained large multimodal large language model to obtain the trained large multimodal large language model and the trained small multimodal large language model.

2. The method for collaboratively training large and small models of a multimodal large language model according to claim 1, characterized in that: The multimodal token is combined with the text token, the pre-trained large multimodal language model is forward-distilled through the pre-trained small multimodal language model, and the projection layer is trained to obtain the trained projection layer, including: The small multimodal large language model generates corresponding text descriptions based on multimodal tokens and text tokens; The projection layer spatially aligns the multimodal tokens with the text tokens to generate aligned multimodal features; The large multimodal language model generates corresponding text descriptions based on the aligned multimodal features; Calculate the first total loss based on the pre-constructed first loss function and the generated text description; The optimization algorithm is used to update the parameters of the projection layer to minimize the first total loss and obtain the trained projection layer.

3. The method for collaboratively training large and small models of a multimodal large language model according to claim 2, characterized in that: The first total loss is calculated based on the pre-constructed first loss function and the generated text description, including: Calculate the difference between the text description generated by the small multimodal large language model and the true label of the text sample to get the first caption loss; Calculate the KL divergence between the probability distribution of the text description generated by the small multimodal large language model and the probability distribution of the text description generated by the large multimodal large language model to obtain the forward KLD loss; The first subtitle loss and the forward KLD loss are input into a pre-constructed first loss function to obtain the first total loss.

4. The method for collaboratively training large and small models of a multimodal large language model according to claim 1, characterized in that: The method uses the trained projection layer and the pre-trained large multimodal large language model to reversely distill the pre-trained small multimodal large language model to obtain the trained large multimodal large language model and the trained small multimodal large language model, including: The small multimodal large language model generates corresponding text descriptions based on multimodal tokens and text tokens; Use the trained projection layer to spatially align the multimodal tokens with the text tokens to generate aligned multimodal features; The large multimodal language model generates corresponding text descriptions based on the aligned multimodal features; Calculate the second total loss based on the pre-constructed second loss function and the generated text description; The parameters of the pre-trained small multimodal large language model and the parameters of the pre-trained large multimodal large language model are updated using an optimization algorithm to minimize the second total loss, thereby obtaining a trained large multimodal large language model and a trained small multimodal large language model.

5. The method for collaboratively training large and small models of a multimodal large language model according to claim 4, characterized in that: The calculating the second total loss based on the pre-constructed second loss function and the generated text description includes: Calculate the difference between the text description generated by the large multimodal language model and the true label of the text sample to obtain the second caption loss; Calculate the KL divergence between the probability distribution of the text description generated by the large multimodal large language model and the probability distribution of the text description generated by the small multimodal large language model to obtain the inverse KLD loss; The second subtitle loss and the inverse KLD loss are input into a pre-constructed second loss function to obtain the second total loss.

6. The method for collaboratively training large and small models of a multimodal large language model according to claim 1, characterized in that: The multimodal token includes a first visual token, a second visual token, and a third visual token, and the multimodal shared encoder includes an image word segmenter, a video word segmenter, an audio word segmenter, and a visual encoder; The step of inputting a multimodal sample into the multimodal shared encoder to generate a multimodal token comprises: The image segmenter divides the image into a plurality of regions to obtain a first image sequence, and the visual encoder encodes the first image sequence to obtain a first visual token; The video segmenter extracts multiple video frames from the video, and divides each video frame into multiple regions to obtain a second image sequence. The visual encoder encodes the second image sequence to obtain a corresponding embedding representation, and the embedding representations of multiple video frames are spliced ​​to obtain a second visual token. The audio word segmenter converts the audio into multiple Mel-spectrograms, and divides each Mel-spectrogram into multiple regions to obtain a third image sequence. The visual encoder encodes the third image sequence to obtain an embedded representation, and concatenates the embedded representations of the multiple Mel-spectrograms to obtain a third visual token.

7. A device for collaborative training of large and small models of a multimodal large language model, characterized in that: include: An acquisition module is used to acquire a multimodal large language model, a multimodal sample and a text sample; wherein the multimodal large language model includes a multimodal shared encoder, a text encoder, a pre-trained small multimodal large language model and a pre-trained large multimodal large language model, the small multimodal large language model and the large multimodal large language model are connected and aligned through a projection layer, and the multimodal sample is a combination of images, videos and audios; A multimodal encoding module, used for inputting multimodal samples into the multimodal shared encoder to generate multimodal tokens; A text encoding module, used for inputting a text sample into the text encoder to obtain a text token; A forward distillation module, for combining the multimodal tokens with the text tokens, forward distilling the pre-trained large multimodal language model through the pre-trained small multimodal language model and training the projection layer to obtain a trained projection layer; The reverse distillation module is used to reversely distill the pre-trained small multimodal large language model by using the trained projection layer and the pre-trained large multimodal large language model to obtain the trained large multimodal large language model and the trained small multimodal large language model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the large and small model collaborative training method of the multimodal large language model is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the large and small model collaborative training method of the multimodal large language model as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the large and small model collaborative training method of the multimodal large language model as claimed in any one of claims 1 to 6 is implemented.

Images