A method and device for size model collaborative training of a multi-modal large language model

Abstract

The application provides a size model cooperative training method and device of a multimodal large language model, the method comprising: inputting image, video, audio and other multimodal samples into a multimodal shared encoder to generate multimodal tokens; inputting text samples into a text encoder to obtain text tokens; combining the multimodal tokens of different modalities with the text tokens, forward distilling a pre-trained large multimodal large language model through a pre-trained small multimodal large language model, and training a projection layer to obtain a trained projection layer, the projection layer being connected between the small and large multimodal large language models; and using the trained projection layer and the pre-trained large multimodal large language model to backward distill the small multimodal large language model, so that the training process simultaneously obtains the trained large and small multimodal large language models. Not only is the cross-modal alignment capability of the large multimodal large language model improved, but also the performance of the small multimodal large language model is enhanced through knowledge distillation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for collaborative training of large and small models of a multimodal large language model. Background Technology

[0002] In recent years, multimodal large-scale pre-trained models have achieved significant breakthroughs in the field of artificial intelligence. These models, by integrating visual and linguistic information, have demonstrated outstanding performance in complex tasks such as image captioning and visual question answering. However, existing multimodal large-language models (MLLMs) typically follow two independent research paths: one is to improve performance by expanding the model size, and the other is to reduce parameters through methods such as pruning to adapt to environments with limited computational resources. The independence of these two paths leads to inefficient training strategies and poor interconnectivity between models, failing to fully utilize the respective advantages of large and small models. Summary of the Invention

[0003] This invention provides a method and apparatus for collaborative training of large and small multimodal language models, addressing the shortcomings of existing multimodal language model training strategies, such as low efficiency, poor inter-model connectivity, and failure to fully utilize the advantages of both large and small models. This invention enables collaborative training of large and small multimodal language models. The technical solution proposed by this invention is as follows:

[0004] In a first aspect, the present invention provides a method for collaborative training of large and small models of a multimodal large language model, comprising:

[0005] Acquire multimodal large language models, multimodal samples, and text samples; wherein, the multimodal large language model includes a multimodal shared encoder, a text encoder, a pre-trained small multimodal large language model, and a pre-trained large multimodal large language model, the small multimodal large language model and the large multimodal large language model are aligned by a projection layer, and the multimodal samples are a combination of images, videos and audio;

[0006] The multimodal samples are input into the multimodal shared encoder to generate multimodal tokens;

[0007] Input the text sample into the text encoder to obtain the text token;

[0008] By combining multimodal tokens with text tokens, the pre-trained large multimodal language model is forward distilled using a pre-trained small multimodal language model, and the projection layer is trained to obtain the trained projection layer.

[0009] Using the trained projection layer and the pre-trained large multimodal language model, the pre-trained small multimodal language model is back-distilled to obtain the trained large multimodal language model and the trained small multimodal language model.

[0010] Optionally, the step of combining multimodal tokens with text tokens, forward distilling the pre-trained large multimodal language model using a pre-trained small multimodal language model, and training the projection layer to obtain the trained projection layer includes:

[0011] A small-scale multimodal large-scale language model generates corresponding text descriptions based on multimodal tokens and text tokens;

[0012] The projection layer spatially aligns the multimodal tokens with the text tokens to generate aligned multimodal features;

[0013] Large-scale multimodal language models generate corresponding text descriptions based on aligned multimodal features;

[0014] Calculate the first total loss based on the pre-constructed first loss function and the generated text description;

[0015] The parameters of the projection layer are updated using an optimization algorithm to minimize the first total loss, resulting in the trained projection layer.

[0016] Optionally, calculating the first total loss based on a pre-constructed first loss function and the generated text description includes:

[0017] The difference between the text description generated by the small multimodal large language model and the real label of the text sample is calculated to obtain the first caption loss;

[0018] The forward KLD loss is obtained by calculating the KL divergence between the probability distribution of text descriptions generated by small multimodal large language models and the probability distribution of text descriptions generated by large multimodal large language models.

[0019] The first subtitle loss and the forward KLD loss are input into a pre-constructed first loss function to obtain the first total loss.

[0020] Optionally, the step of back-distilling the pre-trained small multimodal language model using the trained projection layer and the pre-trained large multimodal language model to obtain the trained large multimodal language model and the trained small multimodal language model includes:

[0021] A small-scale multimodal large-scale language model generates corresponding text descriptions based on multimodal tokens and text tokens;

[0022] The trained projection layer is used to spatially align the multimodal tokens with the text tokens to generate aligned multimodal features.

[0023] Large-scale multimodal language models generate corresponding text descriptions based on aligned multimodal features;

[0024] The second total loss is calculated based on the pre-constructed second loss function and the generated text description;

[0025] The parameters of the pre-trained small multimodal large language model and the pre-trained large multimodal large language model are updated using an optimization algorithm to minimize the second total loss, resulting in the trained large multimodal large language model and the trained small multimodal large language model.

[0026] Optionally, the calculation of the second total loss based on the pre-constructed second loss function and the generated text description includes:

[0027] The difference between the text descriptions generated by the large multimodal language model and the real labels of the text samples is calculated to obtain the second caption loss;

[0028] The KL divergence between the probability distribution of text descriptions generated by large multimodal large language models and the probability distribution of text descriptions generated by small multimodal large language models is calculated to obtain the inverse KLD loss.

[0029] The second subtitle loss and the inverse KLD loss are input into a pre-constructed second loss function to obtain the second total loss.

[0030] Optionally, the multimodal token includes a first visual token, a second visual token, and a third visual token, and the multimodal shared encoder includes an image segmenter, a video segmenter, an audio segmenter, and a visual encoder;

[0031] The step of inputting multimodal samples into the multimodal shared encoder to generate multimodal tokens includes:

[0032] The image segmenter divides the image into multiple regions to obtain the first image sequence, and the visual encoder encodes the first image sequence to obtain the first visual token;

[0033] The video segmenter extracts multiple video frames from the video and divides each video frame into multiple regions to obtain a second image sequence. The visual encoder encodes the second image sequence to obtain the corresponding embedding representation. The embedding representations of multiple video frames are concatenated to obtain a second visual token.

[0034] The audio segmenter converts the audio into multiple Mel spectrograms and divides each Mel spectrogram into multiple regions to obtain a third image sequence. The visual encoder encodes the third image sequence to obtain an embedding representation. The embedding representations of multiple Mel spectrograms are concatenated to obtain a third visual token.

[0035] Secondly, the present invention also provides a multimodal large language model large-scale model co-training device, comprising:

[0036] The acquisition module is used to acquire multimodal large language models, multimodal samples, and text samples. The multimodal large language model includes a multimodal shared encoder, a text encoder, a pre-trained small multimodal large language model, and a pre-trained large multimodal large language model. The small multimodal large language model and the large multimodal large language model are aligned by a projection layer. The multimodal samples are a combination of images, videos, and audio.

[0037] A multimodal encoding module is used to input multimodal samples into the multimodal shared encoder to generate multimodal tokens;

[0038] The text encoding module is used to input text samples into the text encoder to obtain text tokens;

[0039] The forward distillation module is used to combine multimodal tokens with text tokens, forward distill the pre-trained large multimodal language model through a pre-trained small multimodal language model, and train the projection layer to obtain the trained projection layer.

[0040] The backdistillation module is used to backdistill the pre-trained small multimodal language model using the trained projection layer and the pre-trained large multimodal language model to obtain the trained large multimodal language model and the trained small multimodal language model.

[0041] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multimodal large language model co-training method as described in the first aspect above.

[0042] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal large language model co-training method as described in the first aspect above.

[0043] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal large language model co-training method as described in the first aspect above.

[0044] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows:

[0045] This invention provides a method and apparatus for co-training large and small multimodal language models. Through a co-training framework, a large multimodal language model and a small multimodal language model are connected via a projection layer. The lightweight nature of the small multimodal language model enables rapid alignment of multimodal information, thereby improving overall training efficiency. Then, knowledge distillation helps the large multimodal language model better align cross-modal information, while the rich knowledge base of the large multimodal language model enhances the performance of the small multimodal language model. The co-training of the large and small multimodal language models not only improves the cross-modal alignment capability of the large multimodal language model but also enhances the performance of the small multimodal language model through knowledge distillation. This achieves a tight integration between the large and small multimodal language models, with mutually reinforcing capabilities, allowing direct adaptation to various computing resource scenarios.

[0046] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0047] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating the method for collaborative training of large and small models in a multimodal large language model provided by the present invention.

[0050] Figure 2 This is a schematic diagram of the principle framework for collaborative training of large and small multimodal language models provided by the present invention.

[0051] Figure 3 This is a schematic diagram of the collaborative architecture of the large and small multimodal language model provided by the present invention.

[0052] Figure 4 This is a schematic diagram of the collaborative training strategy provided by the present invention.

[0053] Figure 5 This is a schematic diagram of the structure of the multimodal large language model co-training device provided by the present invention.

[0054] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0056] Existing multimodal large language models (MLLMs) mainly fall into two independent research paths: one involves increasing the model size to improve performance, and the other focuses on reducing parameters through pruning and other methods to adapt to environments with limited computing resources. Both large and small models have their own advantages, and collaborative training can produce better results compared to independent training strategies.

[0057] The mainstream approach in MLLM research is to increase model size and provide more data to achieve better scalability. However, as model size increases, the demand for computational resources also increases, making MLLM inference deployment less efficient and more challenging. Therefore, recent research has focused on improving the inference efficiency and deployment ease of MLLM, including methods such as model compression, pruning, and distillation to address these challenges.

[0058] However, these two mainstream research directions are independent and lack integration. Current training strategies are relatively inefficient, with poor connectivity between models, failing to effectively utilize the respective advantages of large and small multimodal language models. In other words, there is a lack of a comprehensive method to leverage the strengths of both to mutually enhance the training process. To address this, this invention proposes a novel collaborative training strategy for large and small models, jointly utilizing these two training methods to develop a more efficient training framework that simultaneously improves the performance of both models. In this collaborative training framework, the large multimodal language model can impart better knowledge to the small multimodal language model through knowledge distillation, while the small multimodal language model can assist the large multimodal language model through multimodal alignment and providing a broader range of multimodal information feature representations than a single model setting.

[0059] The following is combined with Figures 1-6 This invention describes a method and apparatus for collaborative training of large and small models in a multimodal large language model.

[0060] This invention employs a novel collaborative training framework that connects a large multimodal language model and a small multimodal language model through a projection layer. The lightweight nature of the small multimodal language model effectively aligns multimodal information, and knowledge distillation helps the large multimodal language model better align cross-modal information. During the fine-tuning phase, the rich knowledge base of the large multimodal language model is utilized to enhance the performance of the small multimodal language model.

[0061] The training process of the multimodal model is divided into two stages: an alignment stage and a fine-tuning stage. These two stages respectively achieve collaboration between a small multimodal large language model and a large multimodal large language model. In the alignment stage, the small multimodal large language model pre-aligns multimodal information, specifically aligning image, video, and audio multimodal image features with text features, thus helping the large multimodal large language model achieve better image-text alignment. In the fine-tuning stage, the richer knowledge of the large multimodal large language model is utilized through knowledge distillation via Kullback-Leibler divergence (KLD) to guide the learning of the small multimodal large language model.

[0062] Figure 1 A flowchart illustrating the method for collaborative training of large and small models in a multimodal large language model provided by this invention is shown below. The method includes the following:

[0063] Step S101: Obtain the multimodal large language model, multimodal samples, and text samples.

[0064] The aforementioned multimodal large language models include a multimodal shared encoder, a text encoder, a pre-trained small multimodal large language model, and a pre-trained large multimodal large language model. The multimodal samples are combinations of images, videos, and audio, etc.

[0065] Reference Figure 2 As shown, the aforementioned small-scale multimodal large language model has a relatively small parameter size, making it suitable for resource-constrained environments or scenarios requiring rapid deployment. The pre-training method for this small-scale multimodal large language model involves: collecting a large amount of text data as training corpus; performing pre-training using unsupervised or weakly supervised learning tasks; and employing efficient optimization algorithms (such as Adam) and pre-training strategies (such as warm start and progressive training) to accelerate the training process.

[0066] The aforementioned large-scale multimodal language models have a massive parameter scale, enabling them to handle more complex language tasks and exhibit stronger generalization capabilities. The pre-training method for large-scale multimodal language models is similar to that for small-scale multimodal language models, but requires a larger scale of training data. Unsupervised or weakly supervised learning tasks are used for pre-training, and efficient optimization algorithms (such as Adam) and pre-training strategies (such as warm start and progressive training) are employed to accelerate the training process.

[0067] The aforementioned small-scale multimodal language model and large-scale multimodal language model are connected and aligned through a projection layer to achieve feature space transformation and alignment, enabling information transfer and adaptation between language models of different sizes. Linear projection, multi-layer perceptron (MLP), or cross-attention mechanisms can be used as the projection layer.

[0068] Step S102: Input the multimodal sample into the multimodal shared encoder to generate a multimodal token.

[0069] Reference Figure 3 As shown, multimodal samples containing images, videos, and audio are input into the multimodal shared encoder. The multimodal shared encoder processes this input data and generates image tokens representing different modalities (images, video frames, audio features, etc.).

[0070] Step S103: Input the text sample into the text encoder to obtain the text token.

[0071] Reference Figure 3 As shown, a text sample is input into a text encoder, which converts the text into a series of text tokens that capture the semantic and syntactic information in the text.

[0072] Step S104: Combine the multimodal token with the text token, and forward distill the pre-trained large multimodal language model using the pre-trained small multimodal language model and train the projection layer to obtain the trained projection layer.

[0073] In this stage, also known as the alignment stage, a small multimodal large language model is used to align image, video, and audio multimodal image-text features. Visual and linguistic features are quickly aligned through generative learning, and then knowledge distillation is used to assist the large multimodal large language model in cross-modal alignment. During this process, the small multimodal large language model uses forward KLD loss to guide the learning of the large multimodal large language model. The projection layer is trained to help the large multimodal large language model better align multimodal information. Specifically, image tokens generated by the multimodal shared encoder are combined with text tokens generated by the text encoder to form a multimodal input representation. This multimodal input representation is forward-propagated through the pre-trained small multimodal large language model and distilled into the pre-trained large multimodal large language model. In this process, the projection layer is trained to optimize the large multimodal large language model's understanding and processing capabilities of multimodal information. The role of the projection layer is to adjust the feature space output by the small multimodal large language model to better fit the input requirements of the large multimodal large language model.

[0074] Step S105: Using the trained projection layer and the pre-trained large multimodal language model, back-distill the pre-trained small multimodal language model to obtain the trained large multimodal language model and the trained small multimodal language model.

[0075] In this stage, also known as the fine-tuning stage, the knowledge base of the large multimodal language model is used to guide the learning of the small multimodal language model through inverse KLD loss, thereby enhancing the multimodal understanding ability of the small multimodal language model and making its performance stronger. Specifically, the knowledge in the large multimodal language model is backdistilled back into the small multimodal language model using the trained projection layer and the pre-trained large multimodal language model. Through this process, the small multimodal language model can absorb the advantages of the large multimodal language model in processing multimodal information, while maintaining its original lightweight and efficient performance. At the same time, the large multimodal language model also further optimizes its cross-modal processing capabilities through synergy with the projection layer.

[0076] The modal model co-training method provided in this invention connects a large multimodal language model and a small multimodal language model through a projection layer via a co-training framework. Leveraging the lightweight nature of the small multimodal language model, multimodal information can be quickly aligned, thereby improving overall training efficiency. Then, knowledge distillation helps the large multimodal language model better align cross-modal information, while the rich knowledge base of the large multimodal language model enhances the performance of the small multimodal language model. The co-training of the large and small multimodal language models not only improves the cross-modal alignment capability of the large multimodal language model but also enhances the performance of the small multimodal language model through knowledge distillation, enabling it to perform well even in computationally limited environments. This achieves effective processing of multimodal information and the transfer and complementarity of cross-modal knowledge, which is of great significance for improving model performance and generalization ability. The collaborative training method of this invention enables the simultaneous generation of large-scale multimodal language models and small-scale multimodal language models in a single training session. It can directly adapt to various computing resource scenarios (large computing resource cloud scenarios and small computing resource edge scenarios) without the need for successive fine-tuning training and compression distillation, and has broad application prospects.

[0077] Furthermore, through the collaborative training of a multimodal shared encoder, a text encoder, and language models of different sizes, the model can better understand and process information from different modalities, thereby improving overall performance. Forward and backdistillation processes enable knowledge transfer and complementarity between different models, allowing small multimodal large language models to absorb the advantages of large multimodal large language models, while large multimodal large language models further optimize their cross-modal processing capabilities through adjustments to the projection layer. The trained small multimodal large language model maintains its original lightweight and efficient performance, facilitating deployment and use in practical applications. Simultaneously, the large multimodal large language model, through collaborative training, also enhances its flexibility and accuracy in cross-modal processing. In addition, the introduction of multimodal information enables the model to handle more complex and diverse input data, thereby enhancing its generalization ability in practical applications.

[0078] This invention utilizes CLIP-VIT-L / 14 as a multimodal shared encoder and the lightweight language model OPT-125m as a small multimodal large language model. Furthermore, Vicuna-7B is selected as the large multimodal large language model, and the large and small multimodal large language models are connected through projection layers (such as two linear layers) to achieve cross-modal feature alignment. In the collaborative training framework, CLIP-VIT-L / 14, as the multimodal shared encoder, handles the input image, video, and audio multimodal images; OPT-125m, as the small multimodal large language model, handles the input text; and Vicuna-7B, as the large multimodal large language model, performs the final cross-modal task processing. The proposed collaborative training method quickly aligns the multimodal shared encoder and the small multimodal large language model. Subsequently, a distillation strategy is employed to help the large multimodal large language model align multimodal information. During the fine-tuning stage, the multimodal features output by the small multimodal large language model enhance the multimodal understanding capability of the large multimodal large language model. Furthermore, large multimodal language models can impart their rich knowledge base to smaller multimodal language models.

[0079] In an alternative embodiment, refer to Figure 3 As shown, a single encoder is shared across image, video, and audio modalities to achieve better unified multimodal representation and more comprehensive modal alignment. The backbone of this multimodal shared encoder network is a pre-trained visual encoder, completely shared by the image, video, and audio modalities. The difference between these modalities lies in their tokenizer components, hereinafter referred to as the image tokenizer, video tokenizer, and audio tokenizer, respectively. The tokenizer segments the input into a sequence of tokens, which are then used as input to the visual encoder. Correspondingly, the image tokens for the images, videos, and audio are referred to as the first visual token, the second visual token, and the third visual token, respectively.

[0080] Step S102 above, which involves inputting multimodal samples into the multimodal shared encoder to generate a multimodal token, includes:

[0081] S1021, The image segmenter divides the image into multiple regions to obtain a first image sequence, and the visual encoder encodes the first image sequence to obtain a first visual token.

[0082] First, the image segmenter divides the input image into multiple regions. This step involves dividing the image into a fixed-size grid, with each grid region considered a candidate for a feature unit. Subsequently, these segmented image regions (i.e., the first image sequence) are fed into a visual encoder for encoding. The visual encoder, such as the Vision Transformer (ViT) or other convolutional neural network (CNN) based encoders, extracts features from each image region and converts these features into a high-dimensional embedding representation (i.e., the aforementioned first visual token).

[0083] S1022, The video segmenter extracts multiple video frames from the video and divides each video frame into multiple regions to obtain a second image sequence. The visual encoder encodes the second image sequence to obtain the corresponding embedding representation. The embedding representations of multiple video frames are concatenated to obtain a second visual token.

[0084] The video segmenter extracts multiple keyframes or all frames from the video stream. For each video frame, the segmenter performs the same segmentation operation as image processing, dividing the video frame into multiple regions to form a second image sequence. For each second image sequence of a video frame, the visual encoder encodes it separately to generate an embedding representation for each video frame. These embedding representations are then concatenated in chronological order to form a second visual token containing overall video information. This concatenation method preserves the temporal dimension of the video, helping the model understand dynamic changes within the video.

[0085] S1023. The audio segmenter converts the audio into multiple Mel spectrograms and divides each Mel spectrogram into multiple regions to obtain a third image sequence. The visual encoder encodes the third image sequence to obtain an embedded representation and concatenates the embedded representations of multiple Mel spectrograms to obtain a third visual token.

[0086] The audio segmenter converts the audio signal into a series of Mel spectrograms. A Mel spectrogram is a two-dimensional representation of audio that captures the frequency and temporal information of the audio signal. The segmenter then divides each Mel spectrogram into multiple regions, similar to image and video processing, forming a third image sequence. For each Mel spectrogram's third image sequence, the visual encoder also encodes it, generating an embedded representation for each spectrogram. These embedded representations are then concatenated in a specific order (such as chronological order or the order in which the spectrograms were generated) to form a third visual token containing overall audio information.

[0087] This invention, by converting image, video, and audio data into similar image token representations, enables a multimodal shared encoder to more easily achieve cross-modal fusion in subsequent processing stages. This unified representation reduces the complexity of the model processing multimodal data. Thanks to the use of a configurable tokenizer and visual encoder, the processing flow can flexibly adapt to image, video, and audio inputs of different sizes and resolutions. In video processing, by preserving the embedded representations of video frames and concatenating them in chronological order, the model can capture the temporal dimension information in the video. Furthermore, the innovative conversion of audio signals into Mel spectrograms and subsequent image-like processing provides a novel and effective representation of audio data. This representation not only preserves the spectral information of the audio but also further extracts high-level features through processing by the visual encoder.

[0088] Furthermore, since the multimodal shared encoder can handle multiple types of media data and generate a unified token representation, it can be easily combined with the text encoder to build a more powerful multimodal model.

[0089] Unlike traditional visual-only backbone and dual-encoder models, the collaborative framework of this invention comprises a unified multimodal shared encoder and two tightly integrated language models of different sizes. The large and small multimodal language models are essentially mirror images of each other, differing primarily in the number of parameters. This symmetry is advantageous because it allows for efficient training of both models using a single set of infrastructure and optimization techniques. This invention uses two projection layers to map modal features to the semantic space of the LLM. For the large multimodal language model, Vicuna-7B was chosen, while for the small multimodal language model, OPT-125m was selected. This configuration is crucial for validating the effectiveness of the collaborative training method of this invention, aiming to harmonize the small and large multimodal language models. To bridge the significant modal and structural differences between the two language models, thereby mutually enhancing their performance, this invention employs a progressive collaborative training strategy. This strategy begins with pre-training the small multimodal language model, learns synchronous visual and linguistic modalities such as images, videos, and audio through generative learning, and ultimately transitions to comprehensive collaborative training employing knowledge distillation. The collaborative training strategy of this invention consists of two stages: feature alignment and instruction fine-tuning. In the feature alignment stage, the small multimodal large language model quickly aligns visual and linguistic features from other modalities, such as images, videos, and audio, using its relatively small parameter size. Subsequently, this invention employs distillation techniques to assist the large multimodal large language model in alignment. In the fine-tuning stage, the large multimodal large language model leverages its powerful linguistic capabilities to enrich the knowledge of the small multimodal large language model through back-distillation.

[0090] This invention employs generative learning to align OPT125m (small multimodal large language model) and CLIP-vit (multimodal shared encoder), focusing on the embedding of image-text pairs in multimodal languages ​​such as images, videos, and audio. On one hand, because their parameter sizes are comparable, small multimodal large language models can be used more efficiently to align image-text pairs in multimodal languages ​​such as images, videos, and audio. On the other hand, because they share the same labeled embeddings, distillation can be used to better facilitate cross-modal alignment of large multimodal large language models. The existing LAION-400M dataset provides 80 million cleaned data samples. During training, text samples are encoded as Tf using OPT125m, and multimodal samples are extracted as If using CLIP-vit.

[0091] Based on the objective function of BLIP-2, this invention, when training a multimodal pre-trained model, uses an objective function composed of multiple loss terms, including ITC loss for learning the correspondence between images, videos, audio, and text; ITM loss for determining whether images and text match; and ITG loss for optimizing the model's ability to generate text based on multimedia content. These loss terms work together to improve the model's multimodal understanding and generation capabilities. This process allows for the extraction of powerful multimodal representations, making the feature space more closely aligned with the visual encoder. Unlike QFormer, the small multimodal large language model inherently possesses multimodal understanding and reasoning capabilities. It can not only perform caption generation but also perform some QA tasks. Text tags are generated sequentially, guiding the large multimodal large language model and the small multimodal large language model. The small multimodal large language model gains powerful multimodal capabilities through its initial alignment with the multimodal shared encoder. Although the large multimodal large language model has powerful language capabilities, its limited cross-modal alignment capabilities cause its multimodal capabilities to lag behind the small multimodal large language model. Therefore, a small multimodal large language model was chosen to guide a large multimodal large language model. By leveraging the cross-modal alignment of the small multimodal large language model in the first stage, the aim is to enhance the image, video, and audio multimodal image-text understanding capabilities of the large multimodal large language model.

[0092] In the collaborative training framework of this invention, distillation and backdistillation techniques are used to connect large-scale multimodal language models and small-scale multimodal language models, enabling them to learn from each other's strengths and thus improve their performance. This invention considers conditional text generation, and the model is based on probability distributions. , Mid-sampling , Generating responses is a common approach for large, multimodal, large language models to perform tasks. Knowledge distillation (KD) is formalized as an optimization problem aimed at minimizing a fixed teacher model distribution. and by parameters Parameterized student model distribution The differences between them. The standard KD method minimizes the forward KLD:

[0093]

[0094] KL divergence KL[p‖ ]:express and The difference between the two distributions. The smaller the value, the closer the two distributions are.

[0095] : indicates that under the true distribution, given hour The conditional probability.

[0096] : Represents the distribution in the parameterized student model Below, given hour The conditional probability.

[0097] However, knowledge distillation (KD) is particularly suitable for large multimodal language models because these models perform generative tasks where low-capacity student models struggle to replicate complex text generation patterns from teacher or human models. To alleviate these challenges for student models, this invention employs inverse KLD in the second stage. The learning objective of the large multimodal language model involves minimizing the inverse KLD between the distributions of the student and teacher models:

[0098]

[0099] Minimizing the inverse KLD is considered to elicit pattern-seeking behavior in generative modeling, where the model... Assign high probability The salient patterns are identified, while smaller patterns are ignored. This phenomenon has been documented in some studies. Conversely, minimizing the forward KLD leads to... Distribute a large number of probability masses to The zero-probability region leads to the generation of low-quality text in practical applications. However, reverse KLD prioritizes... The main pattern plays a crucial role in ensuring the accuracy and reliability of text generation. Unlike sequence-level knowledge distillation that minimizes forward KLD, minimizing backward KLD does not force... Consistent with teacher distribution All samples in the middle Instead, it encourages student models to generate samples that align with teacher preferences within their capabilities, which is easier to achieve for small, multimodal, large language models.

[0100] The combination of the multimodal token and the text token described in step S104 above, as referred to... Figure 4 As shown, the pre-trained large multimodal language model is forward distilled using a pre-trained small multimodal language model, and the projection layer is trained to obtain the trained projection layer, which specifically includes:

[0101] S1041, Small multimodal large language model generates corresponding text descriptions based on multimodal tokens and text tokens.

[0102] The input to a small multimodal large language model includes multimodal tokens (i.e., the first visual token, second visual token, and third visual token mentioned above) and text tokens. The small multimodal large language model receives these multimodal image tokens and text tokens as input and generates corresponding text descriptions based on them. A small multimodal large language model is typically a model with few parameters and high computational efficiency. It utilizes its own parameters and pre-trained knowledge to fuse and process these multimodal tokens, generating a preliminary text output. The output text description is used for loss calculation and model optimization in subsequent steps.

[0103] S1042. The projection layer spatially aligns the multimodal tokens with the text tokens to generate aligned multimodal features.

[0104] The projection layer receives multimodal tokens and text tokens, aligning these tokens from different modalities spatially to generate a unified feature representation rich in multimodal information. This alignment process involves various techniques such as feature transformation, attention mechanisms, and fusion strategies to ensure that features from different modalities can be compared and combined in the same space. The projection layer outputs the aligned multimodal features, which will serve as input to a large-scale multimodal language model. The alignment process can be found in existing descriptions of projection layers, and will not be elaborated upon here.

[0105] S1043, Large-scale multimodal language models generate corresponding text descriptions based on aligned multimodal features.

[0106] Large-scale multimodal language models (MLMs) receive aligned multimodal features from the projection layer as input. Leveraging their powerful parameters and pre-trained knowledge, MLMs perform in-depth processing and analysis of these features to generate a more accurate, richer, and more fluent text description. Because MLMs typically have more parameters and stronger expressive power, they can capture more detail and semantic information. The output text description from the MLM is used together with the text descriptions generated by smaller MLM models for loss calculation and model optimization in subsequent steps.

[0107] S1044. Calculate the first total loss based on the pre-built first loss function and the generated text description;

[0108] This step requires text descriptions generated by a small multimodal large language model and a large multimodal large language model, as well as the ground truth labels for the text samples. First, the difference between the text descriptions generated by the small multimodal large language model and the ground truth labels is calculated (e.g., using cross-entropy loss), yielding the first caption loss. Then, the KL divergence (forward KLD loss) between the probability distributions of the text descriptions generated by the small and large multimodal large language models is calculated to measure the difference between them. The first caption loss measures the direct difference between the model-generated text and the ground truth text, while the forward KLD loss encourages the distribution of text generated by the small multimodal large language model to converge towards the distribution of the large multimodal large language model, achieving knowledge distillation. Finally, the two loss terms are weighted and summed according to predetermined weights to obtain the first total loss. The calculated first total loss value is output, which will be used for model optimization in subsequent steps.

[0109] S1045. Update the parameters of the projection layer using an optimization algorithm to minimize the first total loss, and obtain the trained projection layer.

[0110] The parameters of the projection layer are updated using optimization algorithms (such as gradient descent, Adam, etc.) to minimize the initial total loss. This update process is iterated multiple times until the loss converges or a preset number of training epochs are reached. The trained projection layer parameters are output, which enable the projection layer to better align multimodal tokens and text tokens, thereby improving the performance of the entire multimodal model. Through this process, an optimized projection layer is obtained, which can more effectively align multimodal tokens and text tokens spatially and generate feature representations rich in multimodal information, thus improving the text generation and understanding capabilities of the multimodal model.

[0111] This invention, through spatial alignment of the projection layer, enables the model to better understand the relationships between different modalities, thereby improving the understanding and processing capabilities of multimodal data. By forward distilling a large multimodal language model using a small multimodal language model, the complexity and computational cost of the model can be reduced while maintaining certain performance, achieving model compression and acceleration. With the participation of the large multimodal language model, the model can generate more accurate, richer, and more fluent text descriptions, improving the quality and diversity of text generation. The model of this invention can process and understand multimedia data from different sources, including still images, dynamic videos, and audio information. This multimodal processing capability allows the model to more comprehensively understand complex scenes in the real world. This method is not only applicable to cross-modal processing of images and text but can also be extended to other modalities such as video and audio, providing technical support for applications such as cross-modal retrieval, generation, and understanding.

[0112] In an optional embodiment, the calculation of the first total loss based on the pre-built first loss function and the generated text description, as described in step S1044 above, specifically includes:

[0113] S201. Calculate the difference between the text description generated by the small multimodal large language model and the real label of the text sample to obtain the first caption loss.

[0114] First, a small multimodal large language model generates a text description based on the input image, video, or audio multimodal image tokens and text tokens. Then, this generated text description is compared with the ground truth labels of the text samples (i.e., Figure 4 The model compares the generated text with the actual text (input prompts). A difference metric, such as cross-entropy loss, is used to calculate the difference between the two, yielding the first caption loss. This loss measures the degree of difference between the model-generated text and the real text.

[0115] S202. Calculate the KL divergence between the probability distribution of text descriptions generated by the small multimodal large language model and the probability distribution of text descriptions generated by the large multimodal large language model to obtain the forward KLD loss.

[0116] A large-scale multimodal language model generates a text description based on multimodal features aligned through a projection layer. The KL divergence (KL divergence) between the probability distributions of the text descriptions generated by the smaller multimodal language model and those generated by the larger model is calculated. KL divergence is a method for measuring the difference between two probability distributions; it asymmetrically measures the information loss of one distribution relative to the other. Forward KL divergence encourages the text distribution generated by the smaller multimodal language model to converge towards the distribution of the larger model, achieving knowledge distillation.

[0117] S203. Input the first subtitle loss and the forward KLD loss into a pre-constructed first loss function to obtain the first total loss.

[0118] During the training phase, as shown in Equation 1, forward distillation facilitates the guidance of the small multimodal large language model on the representation learning of the large multimodal large language model. Probability distribution-guided supervision enables the large multimodal large language model to effectively absorb the multimodal understanding capabilities of the small multimodal large language model. Simultaneously, the parameters of the large multimodal large language model are frozen to prevent catastrophic forgetting. In this phase, only the projection layers inserted into the large multimodal large language model are trained. The total loss in this phase includes the first caption loss and the forward KLD loss. The first caption loss and the forward KLD loss are weighted and summed according to the loss function in Equation 1 to obtain the first total loss.

[0119]

[0120] Here, λ is a balance coefficient, which is 0.3 at this stage.

[0121] This invention directly optimizes the similarity between text generated by a small multimodal large language model and real labels through a first caption loss, which helps improve the accuracy and naturalness of text generation. The forward KLD loss achieves knowledge distillation by encouraging the small multimodal large language model to mimic the output distribution of the large multimodal large language model. This allows the small multimodal large language model to reduce computational complexity and resource consumption while maintaining certain performance. Since the loss function is constructed based on multimodal inputs (images, videos, audio) and text output, the training process enhances the model's understanding and processing capabilities of multimodal data. By combining different loss terms, namely caption loss and KLD loss, the trade-off between generation accuracy and distillation effect can be flexibly adjusted to adapt to different application scenarios and needs. This training strategy is not only applicable to current multimodal models but can also be extended to other types of multimodal tasks or models, facilitating model expansion and transfer.

[0122] In an optional embodiment, step S105 above describes using the trained projection layer and the pre-trained large multimodal language model to back-distill the pre-trained small multimodal language model (see [reference]). Figure 4 As shown), the trained large-scale multimodal language model and the trained small-scale multimodal language model are obtained, including:

[0123] S1051, Small-scale multimodal large language model generates corresponding text descriptions based on multimodal tokens and text tokens.

[0124] The small multimodal large language model receives image tokens and text tokens from different modalities (images, video frames, audio) as input. Based on these inputs, the small multimodal large language model generates corresponding text descriptions. This is a preliminary assessment of the current capabilities of the small multimodal large language model.

[0125] S1052. Use the trained projection layer to spatially align the multimodal tokens and text tokens to generate aligned multimodal features.

[0126] The trained projection layer spatially aligns the multimodal tokens with the text tokens to generate aligned multimodal features. These features fuse information from different modalities and are located in the same space, facilitating subsequent processing.

[0127] S1053, Large-scale multimodal language models generate corresponding text descriptions based on aligned multimodal features.

[0128] The aligned multimodal features are then input into a large multimodal language model. This large multimodal language model leverages its powerful parameters and pre-trained knowledge to perform deep processing on these multimodal features and generate corresponding text descriptions. These text descriptions are typically more accurate, richer, and more fluent than those generated by smaller multimodal language models.

[0129] S1054. Calculate the second total loss based on the pre-built second loss function and the generated text description;

[0130] This step requires text descriptions generated by a small multimodal large language model and a large multimodal large language model, as well as the ground truth labels of the text samples. First, the difference between the text descriptions generated by the large multimodal large language model and the ground truth labels is calculated (e.g., using cross-entropy loss), resulting in the second caption loss. Then, the KL divergence (inverse KLD loss) between the probability distributions of the text descriptions generated by the large multimodal large language model and the small multimodal large language model is calculated to measure the difference between them. The first term, i.e., the second caption loss, measures the difference between the text descriptions generated by the large multimodal large language model and the ground truth labels of the text samples. This helps maintain the large multimodal large language model's ability to fit real text. The second term, i.e., the inverse KLD loss, measures the KL divergence between the probability distributions of the text descriptions generated by the large multimodal large language model and the small multimodal large language model. Unlike S104 above, this encourages the small multimodal large language model to learn from the large multimodal large language model and optimize its output distribution. Finally, these two loss terms are weighted and summed according to predetermined weights to obtain the first total loss. Output the first total loss value calculated, which will be used for model optimization in subsequent steps.

[0131] S1055. Use an optimization algorithm to update the parameters of the pre-trained small multimodal large language model and the pre-trained large multimodal large language model to minimize the second total loss, thereby obtaining the trained large multimodal large language model and the trained small multimodal large language model.

[0132] Optimization algorithms (such as gradient descent, Adam, etc.) are used to update the parameters of the pre-trained small multimodal large language model and (optionally) large multimodal large language model to minimize the second total loss. This process is repeated multiple times until the second total loss converges or a preset number of training epochs are reached.

[0133] In this stage, the invention utilizes the inverse KLD method, using a large multimodal language model as a teacher model to distill a small multimodal language model. Since both the small and large multimodal language models acquire strong multimodal understanding capabilities during the second training stage, the rich knowledge base of the large multimodal language model becomes indispensable. During fine-tuning, the large multimodal language model uses the inverse KLD loss to guide the small multimodal language model to achieve even stronger multimodal capabilities. Except for the multimodal shared encoder, all parameters—namely, the parameters of the large and small multimodal language models and the text encoder—are trainable.

[0134] This invention utilizes back-distillation of large-scale multimodal language models, enabling smaller multimodal language models to learn more advanced language representations and generation capabilities, thereby improving the accuracy and fluency of their text generation. This is particularly important in resource-constrained environments or scenarios requiring rapid inference. Since the entire training process is based on multimodal input, the optimized model (whether a small or large multimodal language model) possesses stronger multimodal understanding capabilities, better able to process and understand data from different modalities.

[0135] In an optional embodiment, the calculation of the second total loss based on the pre-built second loss function and the generated text description, as described in step S1054 above, specifically includes:

[0136] S301. Calculate the difference between the text description generated by the large multimodal language model and the real label of the text sample to obtain the second caption loss.

[0137] A large-scale multimodal language model receives aligned multimodal features as input and generates corresponding text descriptions. The text descriptions generated by the large-scale multimodal language model are compared with the ground truth labels of text samples. A difference metric, such as cross-entropy loss, is used to calculate the difference between the two, resulting in a second caption loss. This loss measures the similarity or difference between the text generated by the large-scale multimodal language model and the ground truth text, helping to maintain the large-scale multimodal language model's ability to fit ground truth text.

[0138] S302. Calculate the KL divergence between the probability distribution of text descriptions generated by large multimodal large language models and the probability distribution of text descriptions generated by small multimodal large language models to obtain the inverse KLD loss.

[0139] The KL divergence between the probability distributions of text descriptions generated by a large multimodal language model and those generated by a small multimodal language model is calculated, but the direction is from the small multimodal language model to the large multimodal language model; hence, it is called the inverse KLD loss. This loss encourages the output distribution of the small multimodal language model to converge towards the output distribution of the large multimodal language model, thereby achieving efficient knowledge distillation.

[0140] S303. Input the second subtitle loss and the inverse KLD loss into a pre-constructed second loss function to obtain the second total loss.

[0141] A second loss function is predefined. This function takes the second subtitle loss and the inverse KLD (reKLD) loss as inputs and combines them into a single loss value, i.e., the second total loss, according to certain rules such as weighted summation. The second subtitle loss and the inverse KLD loss are input into the pre-constructed second loss function, and the second total loss is calculated according to the function definition.

[0142]

[0143] Here, λ is a balance coefficient, which is 0.3 at this stage.

[0144] By adjusting the weights of each loss term in the second loss function, the optimization levels of large-scale multimodal language models and small-scale multimodal language models can be balanced according to the specific task requirements, thereby obtaining the best model performance.

[0145] The aforementioned inverse KLD loss encourages the smaller multimodal large language model to learn the output distribution of the larger multimodal large language model, thereby absorbing its high-level language representation and generation capabilities. This helps improve the accuracy and fluency of the smaller multimodal large language model in text generation tasks. The second subtitle loss ensures that the larger multimodal large language model does not lose its ability to fit real text during distillation. By optimizing this loss, the larger multimodal large language model can continue to maintain its strong generation capabilities. The entire process realizes knowledge transfer from the larger multimodal large language model to the smaller multimodal large language model, enabling the smaller multimodal large language model to achieve performance close to that of the larger multimodal large language model while maintaining lower computational complexity. Since the distillation process is based on multimodal input, the optimized model may have stronger multimodal understanding capabilities and be able to better handle and understand data from different modalities. Moreover, by optimizing both models simultaneously, training data can be utilized more effectively, training time can be reduced, and the model's generalization ability can be improved.

[0146] Through the collaborative training strategy of this invention, the large multimodal language model benefits from the more detailed and information-rich multimodal semantic features generated by the small multimodal language model, while the small multimodal language model benefits from the knowledge guidance of the large multimodal language model, thus mutually improving performance. Furthermore, the small multimodal language model of this invention can be directly deployed for inference without requiring new fine-tuning training for each task.

[0147] This invention addresses the shortcomings of independent training strategies in existing technologies by proposing a novel collaborative framework. This framework enables collaborative training of a micro (i.e., a small multimodal large language model) and a large multimodal large language model (i.e., a large multimodal large language model). It connects the large and small multimodal large language models via a projection layer, utilizing a collaborative training strategy to leverage the strengths of each model and improve their performance. Compared to previous methods, this framework offers several advantages: it strengthens the connection between the large and micro models, improving training efficiency. In the collaborative training strategy, it effectively aligns image-text features by utilizing the parameter efficiency of the small multimodal large language model. Knowledge distillation helps the large multimodal large language model better align cross-modal information. During the fine-tuning phase, it leverages the extensive knowledge base of the large multimodal large language model to enhance the performance of the small multimodal large language model. Through the collaborative training method, it achieves a tight integration of the large and small multimodal large language models, mutually enhancing each other's capabilities and allowing direct adaptation to various computational resource scenarios.

[0148] The following describes the multimodal large language model large-scale model co-training device provided by the present invention. The multimodal large language model large-scale model co-training device described below and the multimodal large language model large-scale model large-scale model co-training method described above can be referred to in correspondence.

[0149] Reference Figure 5 As shown, the multimodal large language model co-training device provided by the present invention includes:

[0150] The acquisition module 401 is used to acquire multimodal samples and text samples of the multimodal large language model; wherein, the multimodal large language model includes a multimodal shared encoder, a text encoder, a pre-trained small multimodal large language model and a pre-trained large multimodal large language model, the small multimodal large language model and the large multimodal large language model are aligned by a projection layer, and the multimodal samples are a combination of images, videos and audio;

[0151] The multimodal encoding module 402 is used to input multimodal samples into the multimodal shared encoder to generate multimodal tokens;

[0152] The text encoding module 403 is used to input text samples into the text encoder to obtain text tokens;

[0153] Forward distillation module 404 is used to combine multimodal tokens with text tokens, forward distill the pre-trained large multimodal language model through a pre-trained small multimodal language model, and train the projection layer to obtain the trained projection layer.

[0154] The backdistillation module 405 is used to backdistill the pre-trained small multimodal language model using the trained projection layer and the pre-trained large multimodal language model to obtain the trained large multimodal language model and the trained small multimodal language model.

[0155] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540. The processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions from the memory 530 to execute a multimodal large language model co-training method.

[0156] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0157] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the multimodal large language model co-training method provided by the above methods.

[0158] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for co-training large and small models of a multimodal large language model provided by the methods described above.

[0159] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0160] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for collaborative training of large and small models in a multimodal large language model, characterized in that, include: Acquire multimodal large language models, multimodal samples, and text samples; wherein, the multimodal large language model includes a multimodal shared encoder, a text encoder, a pre-trained small multimodal large language model, and a pre-trained large multimodal large language model, the small multimodal large language model and the large multimodal large language model are aligned by a projection layer, and the multimodal samples are a combination of images, videos and audio; The multimodal samples are input into the multimodal shared encoder to generate multimodal tokens; Input the text sample into the text encoder to obtain the text token; Multimodal tokens are combined with text tokens. A pre-trained small multimodal large language model is used to forward distill the pre-trained large multimodal large language model, and a projection layer is trained to obtain the trained projection layer. An optimization algorithm is used to update the parameters of the projection layer to minimize a first total loss, resulting in the trained projection layer. The first total loss is determined as follows: the difference between the text description generated by the small multimodal large language model and the true label of the text sample is calculated to obtain a first caption loss; the KL divergence between the probability distribution of the text description generated by the small multimodal large language model and the probability distribution of the text description generated by the large multimodal large language model is calculated to obtain a forward KLD loss; the first caption loss and the forward KLD loss are input into a pre-constructed first loss function to obtain the first total loss. Using the trained projection layer and the pre-trained large multimodal language model, the pre-trained small multimodal language model is back-distilled to obtain the trained large multimodal language model and the trained small multimodal language model. The process of back-distilling the pre-trained small multimodal language model using the trained projection layer and the pre-trained large multimodal language model to obtain the trained large multimodal language model and the trained small multimodal language model includes: A small-scale multimodal large-scale language model generates corresponding text descriptions based on multimodal tokens and text tokens; The trained projection layer is used to spatially align the multimodal tokens with the text tokens to generate aligned multimodal features. Large-scale multimodal language models generate corresponding text descriptions based on aligned multimodal features; The second total loss is calculated based on the pre-constructed second loss function and the generated text description; The parameters of the pre-trained small multimodal large language model and the pre-trained large multimodal large language model are updated using an optimization algorithm to minimize the second total loss, resulting in the trained large multimodal large language model and the trained small multimodal large language model. The second total loss is determined as follows: the difference between the text description generated by the large multimodal language model and the true label of the text sample is calculated to obtain the second caption loss; the KL divergence between the probability distribution of the text description generated by the large multimodal language model and the probability distribution of the text description generated by the small multimodal language model is calculated, with the direction from the small multimodal language model to the large multimodal language model, to obtain the inverse KLD loss; the second caption loss and the inverse KLD loss are input into a pre-constructed second loss function to obtain the second total loss; The process of combining multimodal tokens with text tokens, forward distilling the pre-trained large multimodal language model using a pre-trained small multimodal language model, and training the projection layer to obtain the trained projection layer includes: A small-scale multimodal large-scale language model generates corresponding text descriptions based on multimodal tokens and text tokens; The projection layer spatially aligns the multimodal tokens with the text tokens to generate aligned multimodal features; Large-scale multimodal language models generate corresponding text descriptions based on aligned multimodal features; Calculate the first total loss based on the pre-constructed first loss function and the generated text description; The parameters of the projection layer are updated using an optimization algorithm to minimize the first total loss, resulting in the trained projection layer. The multimodal tokens include a first visual token, a second visual token, and a third visual token; the multimodal shared encoder includes an image segmenter, a video segmenter, an audio segmenter, and a visual encoder. The step of inputting multimodal samples into the multimodal shared encoder to generate multimodal tokens includes: The image segmenter divides the image into multiple regions to obtain the first image sequence, and the visual encoder encodes the first image sequence to obtain the first visual token; The video segmenter extracts multiple video frames from the video and divides each video frame into multiple regions to obtain a second image sequence. The visual encoder encodes the second image sequence to obtain the corresponding embedding representation. The embedding representations of multiple video frames are concatenated to obtain a second visual token. The audio segmenter converts the audio into multiple Mel spectrograms and divides each Mel spectrogram into multiple regions to obtain a third image sequence. The visual encoder encodes the third image sequence to obtain an embedding representation. The embedding representations of multiple Mel spectrograms are concatenated to obtain a third visual token.

2. A multimodal large language model large-scale model co-training device, applicable to the multimodal large language model large-scale model large-scale model co-training method of claim 1, characterized in that, include: The acquisition module is used to acquire multimodal large language models, multimodal samples, and text samples. The multimodal large language model includes a multimodal shared encoder, a text encoder, a pre-trained small multimodal large language model, and a pre-trained large multimodal large language model. The small multimodal large language model and the large multimodal large language model are aligned by a projection layer. The multimodal samples are a combination of images, videos, and audio. A multimodal encoding module is used to input multimodal samples into the multimodal shared encoder to generate multimodal tokens; The text encoding module is used to input text samples into the text encoder to obtain text tokens; A forward distillation module is used to combine multimodal tokens and text tokens, forward distill the pre-trained large multimodal language model using a pre-trained small multimodal language model, and train the projection layer to obtain the trained projection layer. An optimization algorithm is used to update the parameters of the projection layer to minimize a first total loss, resulting in the trained projection layer. The first total loss is determined as follows: calculating the difference between the text description generated by the small multimodal language model and the true label of the text sample to obtain a first caption loss; calculating the KL divergence between the probability distribution of the text description generated by the small multimodal language model and the probability distribution of the text description generated by the large multimodal language model to obtain a forward KLD loss; and inputting the first caption loss and the forward KLD loss into a pre-constructed first loss function to obtain the first total loss. The backdistillation module is used to backdistill a pre-trained small multimodal language model using a trained projection layer and a pre-trained large multimodal language model to obtain a trained large multimodal language model and a trained small multimodal language model. The backdistillation process using the trained projection layer and the pre-trained large multimodal language model to obtain the trained large multimodal language model and the trained small multimodal language model includes: the small multimodal language model generating corresponding text descriptions based on multimodal tokens and text tokens; spatially aligning the multimodal tokens and text tokens using the trained projection layer to generate aligned multimodal features; the large multimodal language model generating corresponding text descriptions based on the aligned multimodal features; and finally, based on a pre-constructed second loss function and the generated text descriptions... The process involves calculating a second total loss; updating the parameters of the pre-trained small multimodal large language model and the pre-trained large multimodal large language model using an optimization algorithm to minimize the second total loss, resulting in a trained large multimodal large language model and a trained small multimodal large language model; the second total loss is determined by calculating the difference between the text description generated by the large multimodal large language model and the true label of the text sample, resulting in a second caption loss; calculating the KL divergence between the probability distribution of the text description generated by the large multimodal large language model and the probability distribution of the text description generated by the small multimodal large language model, with the direction from the small multimodal large language model to the large multimodal large language model, resulting in an inverse KLD loss; and inputting the second caption loss and the inverse KLD loss into a pre-constructed second loss function to obtain the second total loss.

3. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for co-training large and small models of a multimodal large language model as described in claim 1.

4. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for co-training large and small models of a multimodal large language model as described in claim 1.

5. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for co-training large and small models of a multimodal large language model as described in claim 1.

Images