A multi-modal knowledge distillation method
By extracting the multimodal semantic structure knowledge of the teacher model and introducing a cross-modal fusion coding module and an adaptive gradient weight mechanism, the problems of insufficient utilization of teacher knowledge and static and coarse weight allocation in existing methods are solved, and efficient cross-modal retrieval of lightweight student models is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 湖南工商大学
- Filing Date
- 2026-06-11
- Publication Date
- 2026-07-14
AI Technical Summary
Existing multi-teacher knowledge distillation methods for text and image suffer from problems such as a single dimension of teacher knowledge utilization, a simple modal feature fusion mechanism, and a static and coarse teacher weight allocation method, resulting in insufficient cross-modal knowledge transfer and high model deployment costs.
By extracting knowledge of image-text similarity, visual similarity, and text similarity from the teacher model, a lightweight student model is constructed. A cross-modal fusion coding module and an adaptive gradient weight mechanism are introduced for mask distillation training, thereby enhancing the cross-modal interaction capability and adaptability of the student model.
While reducing computational costs, it significantly improves the accuracy of image and text retrieval and the robustness of the model, and enhances the cross-modal semantic alignment capability and training stability of the student model.
Smart Images
Figure CN122389997A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of knowledge distillation technology, and in particular to a multimodal knowledge distillation method. Background Technology
[0002] With the rapid development of applications such as search engines, e-commerce product retrieval, and image and stock image retrieval, the demand for image retrieval based on natural language description is increasing. Image-text retrieval, as a typical task in cross-modal retrieval, establishes a correspondence between image content and text semantics, enabling users to efficiently obtain target image information using concise natural language. In recent years, multimodal pre-trained large-scale models, through joint training on large-scale image-text pairs, have significantly improved the semantic alignment capability between images and text, providing strong technical support for the aforementioned applications.
[0003] Existing methods are mostly based on large-scale image-text pre-trained models, such as image-text contrast learning models with dual encoder structures. These models map visual and text features into a unified semantic embedding space and use similarity constraints to achieve cross-modal semantic alignment. However, such models typically rely on large-scale parameters and computational resources, resulting in high training and inference costs, making them difficult to deploy directly in resource-constrained scenarios.
[0004] To reduce model complexity and improve deployment efficiency, knowledge distillation techniques have been introduced into the lightweighting process of text-image pre-trained models. Existing methods typically employ single-teacher or multi-teacher distillation strategies, using the prediction distribution output by the teacher model as soft labels to guide the student model's learning.
[0005] However, existing multi-teacher knowledge distillation methods using text and images still have the following significant shortcomings: The teacher's knowledge utilization dimension is too narrow: most methods only distill from the image-text similarity output layer of the teacher model, without making full use of the structured semantic information inside the teacher model in the visual and text modalities, resulting in insufficient cross-modal knowledge transfer; The modal feature fusion mechanism is simple: existing models mostly rely on the global feature vectors output by the visual and text encoders to calculate cosine similarity, lacking deep interaction mechanisms between modalities, and making it difficult to capture the complex correspondence between local regions of images and fine-grained semantics of text. The teacher weight allocation method is static and crude: In multi-teacher distillation, fixed weights, equal weighting, or simple weighting based on confidence are usually adopted, without considering the dynamic contribution differences of the teacher model under different samples and different semantic spaces, thus introducing irrelevant or redundant knowledge to interfere with the student model learning. Summary of the Invention
[0006] To address the above problems, this invention provides a multimodal knowledge distillation method, which aims to fully mine the multimodal semantic structure knowledge of the teacher model, enhance the cross-modal interaction capability of the student model, and achieve adaptive gradient weight allocation to improve the cross-modal retrieval performance of the lightweight student model.
[0007] This invention provides a multimodal knowledge distillation method, comprising: S1. Obtain the image-text pair dataset and preprocess the image-text pairs using a pre-trained large model to obtain a high-quality image-text pair dataset. S2, input the high-quality image-text pair dataset into the pre-trained teacher multimodal model, and extract the image-text similarity knowledge, visual similarity knowledge and text similarity knowledge of the teacher model; S3, construct a lightweight student model, and use a high-quality image and text dataset to train the student model by mask distillation; S4 uses a high-quality unmasked image-text pair dataset and fine-tunes the student model using only the image-text contrast learning loss as the optimization objective to obtain the final student model.
[0008] Furthermore, S1 specifically includes: S11, use the image text description generation model to generate text for each original image in the image-text pair dataset, and together with the original text, form a candidate text set; S12 uses an image-text matching model to calculate the matching score between each original image and each text in the candidate text set, selects the text with the highest score as the optimal text description, and forms a high-quality image-text pair dataset with the original image.
[0009] By introducing an image-text description generation model and an image-text matching model, the original image-text pairs are subjected to dual screening. This effectively removes noisy samples and completes semantically inconsistent descriptions, improving the quality and consistency of the image-text pair dataset. This provides more reliable training samples for the subsequent distillation process, thereby improving the convergence stability and final retrieval accuracy of the student model.
[0010] Furthermore, S2 specifically includes: The image-text pairs are input into the visual encoder and text encoder of the teacher model respectively to obtain the original visual feature vector and the original text feature vector. After being projected into the joint semantic space and normalized, the image-text cosine similarity is calculated to obtain the image-text similarity knowledge of the teacher model. Visual similarity knowledge is obtained by calculating the cosine similarity between different images using normalized visual feature vectors. By calculating the cosine similarity between different texts using the normalized text feature vectors, we can obtain text similarity knowledge.
[0011] By extracting image-text similarity knowledge, visual similarity knowledge, and text similarity knowledge from the teacher model, we fully explored the fine-grained semantic structure information of different modalities within the teacher model, enriching the sources of distilled knowledge. This enabled the student model to align with the teacher model at the semantic distribution level within the modality, thereby enhancing students' ability to model image-text semantic structures.
[0012] Furthermore, the student model includes: A visual encoder is used to extract visual features from an image; A text encoder is used to extract semantic features from text. A cross-modal fusion coding module is used to enable bidirectional interaction between visual and textual data; The similarity calculation module is used to calculate image-text similarity, visual similarity, and text similarity.
[0013] By configuring a visual encoder, a text encoder, a cross-modal fusion encoding module, and a similarity calculation module for the student model, a lightweight network structure with deep modal interaction capabilities was constructed. The cross-modal fusion encoding module compensates for the insufficient interaction of the dual-encoder model, enabling the student model to actively learn fine-grained correspondences between images and text during distillation. This provides a stronger representational foundation for subsequent mask distillation training, thereby improving the accuracy and robustness of cross-modal retrieval.
[0014] Furthermore, the visual encoder adopts a hierarchical semantic grouping visual encoding structure, which includes multiple encoding stages. In the encoding stage, learnable semantic aggregation vectors are introduced, and the current visual feature vector is gradually aggregated into the corresponding number of group vectors through grouping aggregation operations.
[0015] The above structural design enables the model to gradually abstract high-level semantic object representations from low-level image patches, effectively breaking the limitations of regular grid representation, enhancing the structuring and semantic capabilities of visual features, and providing richer and more discriminative visual representations for student models.
[0016] Furthermore, the grouping aggregation operation includes: Calculate the correlation matrix between visual feature vectors and semantic aggregation vectors; Hard-valued semantic grouping relationships are generated based on the correlation matrix, so that each visual feature vector belongs to only one semantic aggregation vector; Based on the hard-value semantic grouping relationship, visual feature vectors belonging to the same semantic aggregation vector are weighted and aggregated to obtain the visual feature vector of the next layer.
[0017] By calculating the correlation matrix, generating hard-valued semantic grouping relationships, and performing weighted aggregation based on the grouping relationships, it is ensured that each visual feature vector belongs to only one semantic aggregation vector, avoiding semantic confusion caused by ambiguous attribution, extracting independent semantic regions in the image more accurately, improving the efficiency and accuracy of semantic aggregation, and thus generating more discriminative high-level visual features.
[0018] Furthermore, the cross-modal fusion coding module is specifically used for: The visual feature vectors output by the visual encoder and the text feature vectors output by the text encoder are projected onto the query, key, and value spaces, respectively. The query vector of the visual feature vector is used to retrieve the key-value pairs of the text feature vector, and a visual-to-text context vector is generated. The query vector of the text feature vector is used to retrieve the key-value pairs of the visual feature vector, and a text-to-visual context vector is generated. The generated context vector is fed back to the encoder of the corresponding modality through residual connections for updating, and multiple layers of cross-modal fusion coding modules are stacked to obtain the final visual representation and the final text representation that aggregate global semantic information.
[0019] The cross-modal fusion encoding module enables visual and textual representations to enhance each other in multi-round interactions, effectively capturing the complex correspondence between local regions of images and fine-grained semantics of text, significantly improving the cross-modal semantic alignment accuracy of the lightweight student model, and making up for the shortcomings of the dual encoder interaction.
[0020] Furthermore, the step of training the student model using mask distillation on a high-quality image and text dataset specifically includes: Randomly mask the images in the high-quality image-text pair dataset to construct a masked image-text pair dataset, and input it into the student model to calculate the image-text similarity, visual similarity, and text similarity of the student model. Calculate the distillation loss of teacher-student similarity based on the image-text similarity of the teacher and student models; Calculate the teacher-student multimodal similarity distillation loss based on visual and textual similarity of teacher and student models; An adaptive weighting mechanism for multimodal gradient coordination is introduced to weight and fuse the visual similarity and text similarity of the teacher model to obtain a multimodal fusion similarity. The multimodal fusion similarity distillation loss is then calculated based on the image-text similarity of the student model. Construct a text-image comparison learning loss based on text-image matching and real-label; The weighted sum of the above four types of losses is used as the total loss for training the student model.
[0021] The training strategy described above enables the student model to learn simultaneously from the teacher's text-image similarity, intra-modal similarity, and fusion similarity. It also maintains consistency with the real labels through contrastive learning, avoiding the learning of the teacher's erroneous knowledge. This reduces computational costs while improving distillation performance and model robustness.
[0022] Furthermore, the random masking process for images in the high-quality image-text dataset specifically includes: The image is divided into multiple image blocks, a binary mask vector is constructed, and a preset proportion of image blocks are randomly set as the mask, while the remaining image blocks are retained for encoding, resulting in a masked image-text pair dataset.
[0023] By performing random masking on the images, the number of image blocks input to the visual encoder in each iteration can be significantly reduced, thereby reducing memory usage and computational load. This allows for an increase in training batch size and training epochs with limited resources, thus improving training efficiency.
[0024] Furthermore, the adaptive weighting mechanism for multimodal gradient coordination specifically includes: The cross-entropy loss between the visual similarity and text similarity of the teacher model and the image-text similarity of the student model is calculated separately to obtain the visual gradient and text gradient of the student model. We introduce visual weight coefficients and text weight coefficients, and determine the optimal weight allocation by minimizing the weighted gradient norm. The visual similarity and textual similarity of the teacher model are weighted and fused using the optimal weights to obtain the multimodal fused similarity.
[0025] The adaptive weighting mechanism for multimodal gradient coordination effectively mitigates gradient conflicts by calculating the cross-entropy gradient of visual and textual similarity targets and dynamically allocating modal weights by minimizing the weighted gradient norm. This mechanism eliminates the need for manually pre-defined prior rules and adaptively balances the optimization contributions of each modality based on the current samples, avoiding the dominance of single-modal gradients in parameter updates. This accelerates the convergence speed of the student model and enhances the synergy of multimodal knowledge fusion and the generalization performance of the final model.
[0026] Compared with existing technologies, the advantages of this invention are as follows: By explicitly distilling the modal semantic structure within the teacher model, introducing a cross-modal fusion coding module, and employing a gradient-coordinated adaptive weight mechanism, this invention systematically overcomes the shortcomings of existing methods, such as single knowledge sources, insufficient modal interaction, and static and coarse weight allocation. This invention fully explores the fine-grained semantic relationships within the visual and text modalities, enhances the deep cross-modal interaction capabilities of the student model, and achieves dynamic adaptive adjustment of distillation weights. Thus, while maintaining lightweight inference costs, it significantly improves the accuracy and training stability of image and text retrieval. Attached Figure Description
[0027] To more clearly illustrate the technical solutions in the embodiments of this drawing or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this drawing. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.
[0028] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a flowchart of the image-text pair data preprocessing process of the present invention; Figure 3 This is a flowchart of the mask distillation learning process of the present invention; Figure 4 This is a flowchart of the small-round non-mask learning process of the present invention. Detailed Implementation
[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments provided by this invention without inventive effort are within the scope of protection of this invention.
[0030] This invention provides a multimodal knowledge distillation method, such as... Figure 1 As shown, it specifically includes: S1. Obtain the image-text pair dataset and preprocess the image-text pairs using a pre-trained large model to obtain a high-quality image-text pair dataset.
[0031] S11, such as Figure 2 As shown, the original image-text pair dataset is obtained. ,in, For the original image, The text description is the original image, and n is the total number of image-text pairs; Image-text description generation models (such as the multimodal base model CoCa) are used to generate text for each original image in the image-text pair dataset. Generative models generate text : ; in, Image-to-text function for generating image-to-text description models; Generate parameters for the model to describe the image with text; Original text Text generated by the model Together they constitute the candidate text set .
[0032] S12, use the image-text matching model to calculate the matching score between each original image and each text in the candidate text set, and select the text with the highest score as the optimal text description. : ; in, This is the softmax scoring function for the image-text matching model; These are the parameters for the image-text matching model; T represents a text description from the candidate text set; This represents the textual description that causes the fractional function to reach its maximum value; Original image With optimal text description Composing a high-quality image-text pair dataset .
[0033] S2, input the high-quality image-text pair dataset into the pre-trained teacher multimodal model, and extract the image-text similarity knowledge, visual similarity knowledge and text similarity knowledge of the teacher model.
[0034] Extraction of image-text similarity knowledge: such as Figure 3 As shown, a batch of preprocessed images The visual encoder of the input teacher model The original visual feature vector is obtained. : ; in, These are the parameters of the visual encoder for the teacher model; N is the batch size; The spatial dimension of the original visual feature vector.
[0035] A batch of preprocessed text Text encoder for input teacher model The original feature vector of the text is obtained. : ; in, These are the parameters of the text encoder for the teacher model; The spatial dimension of the original feature vector of the text.
[0036] Will and Mapped to the same graph-text joint semantic space via projection matrix: ; ; in, , The learnable linear projection matrix is used to respectively... The original visual feature vector of dimension and The original text feature vector of dimension is linearly mapped to... In the dimensional graph-text joint semantic space; , These are the mapped visual feature vector and text feature vector, respectively.
[0037] Will and Perform L2 normalization while preserving dimensions. Dimensional invariance: ; ; in, , These are the normalized visual feature vector and text feature vector, respectively; It is an L2 norm.
[0038] The image-text similarity knowledge is obtained by calculating the image-text cosine similarity of the teacher model. Since the magnitude of the normalized image-text feature vectors is 1, calculating the image-text similarity only requires a dot product operation: ; in, This is the image-text similarity matrix; for The transpose of .
[0039] Extraction of visual and text similarity knowledge: Visual similarity and text similarity are calculated using normalized visual feature vectors and text feature vectors, respectively. ; ; in, , These are the visual similarity matrix and the text similarity matrix, respectively. for The transpose of .
[0040] S3: Construct a lightweight student model and train it using mask distillation on a high-quality image and text dataset.
[0041] Specifically, a student model with the same task format as the teacher model but with a smaller parameter scale is constructed. The student model includes a student visual encoder, a student text encoder, a cross-modal fusion encoding module, and a similarity calculation module.
[0042] like Figure 3 As shown, images in the high-quality image-text pair dataset are randomly masked to construct a masked image-text pair dataset. Specifically, the images are... Divide into m image blocks Construct the corresponding binary mask vector ,in This indicates that the image is preserved. This means that the image is masked. A preset proportion of image blocks are randomly selected as the mask, while the remaining image blocks are retained for encoding, resulting in a masked image. , and corresponding text Composition of masked image-text pairs dataset Input the masked image-text pair dataset into the student model, and calculate the image-text similarity, visual similarity, and text similarity of the student model.
[0043] The student visual encoder employs a hierarchical semantic grouping visual coding structure. The encoder consists of 12 layers: the first 9 layers are used for layer-by-layer semantic aggregation coding, and the last 3 layers are used for cross-modal fusion coding. The mask image is then processed. Initial visual vectors are obtained through linear mapping. : ; in, For image embedding matrix; This is the i-th visual vector of the first layer.
[0044] The visual encoder adopts the GroupViT encoder architecture. To achieve hierarchical semantic grouping, 64 and 8 learnable semantic aggregation vectors are introduced in layers 1 and 4, respectively. , Representing the Layer coding stage, represent The number of semantic aggregation vectors in the layer. Subsequently, in layers 3 and 6, the current visual vector is progressively aggregated into the corresponding number of group vectors through group aggregation operations.
[0045] Specifically, in the encoding stage, visual vectors are... and semantic aggregation vector The encoded visual vector is obtained by concatenating the input GroupViT encoding layer. and semantic aggregation vector : ; in, Indicates a splicing operation; This represents a GroupViT encoding block, which will be encoded as follows: The two should be distinguished.
[0046] For group clustering operations, the correlation matrix between visual vectors and semantic aggregation vectors is calculated at layers 3 and 6. : ; in, Indicates the first The similarity value between the i-th semantic aggregation vector and the j-th visual vector in the layer; This represents the learnable projection matrix used to map semantic aggregation vectors to the query space; This represents the learnable projection matrix used to map visual vectors to the key space; exp It is a natural exponential function.
[0047] Hard-valued semantic grouping relationships are generated based on the correlation matrix, ensuring that each visual feature vector belongs to only one semantic aggregation vector: ; ; in, Indicates the first The hardware matrix after grouping and assignment during the layer grouping stage; Indicates the first The similarity value between the k-th semantic aggregation vector and the j-th visual vector in the layer; This operation represents the index of the maximum similarity value along the dimension of the semantic aggregation vector, used to determine the optimal semantic aggregation vector to which the visual vector belongs; This ensures that each visual vector is assigned only to the semantic aggregation vector with the highest relevance, thus satisfying the grouping constraint. This indicates that each visual vector belongs to only one semantic group.
[0048] Based on the hard-value semantic grouping relationship, visual feature vectors belonging to the same semantic aggregation vector are weighted and aggregated to obtain the visual feature vector of the next layer. : ; in, Indicates the first The i-th visual vector in layer +1; The values of the visual vectors can be projected onto a learnable linear matrix; The output projection of the fused features can be learned as a linear matrix.
[0049] After grouping and aggregating at layer 6, eight visual vectors are obtained. A cls vector representing global visual information is added to these vectors before subsequent encoding. Finally, after the first nine layers of encoding and grouping clustering, nine visual vectors are obtained. ,in It represents global visual information.
[0050] The student text encoder adopts a Transformer architecture, consisting of 12 Transformer blocks. The first 9 layers use regular text encoding, and the last 3 layers use cross-modal fusion encoding. Each layer contains a multi-head self-attention mechanism and a feedforward neural network, and finally connects to the next layer through residual connections.
[0051] Specifically, for the input text, first add to the beginning of the sequence... Labels are then used to perform word embeddings to obtain sequences. The sequence is embedded through a text embedding matrix. Mapped to initial text vector , This indicates the number of vectors. To preserve the text's order information, learnable positional codes are added to the embedding vectors. ,Right now .
[0052] The operations at each level can be represented as: ; in, It is the first The output of the -1st layer Transformer block is also the... Input to the Transformer block; This indicates a Transformer block.
[0053] The text is encoded using 9 layers of Transformer to obtain the final text vector. ,in Represents global information about the text.
[0054] The cross-modal fusion coding module is used to enable bidirectional interaction between visual and textual data.
[0055] Specifically, the visual vectors output by the visual encoder and the text vectors output by the text encoder are projected onto the query, key, and value spaces, respectively: ; ; ; in, , and These are the projection weight matrices for the query, key, and value of the visual modality, respectively. , and These are the projection results of the visual vectors into the query, key, and value spaces, respectively. , and These are the query, key, and value projection weight matrices for the text modality, respectively. , and These are the projection results of the text vector into the query, key, and value spaces, respectively.
[0056] Projection of visual vectors into the query space Using the text vector as a query carrier, we retrieve semantic information from the text vector, and simultaneously use the projection result of the text vector onto the query space. To retrieve spatial information within the visual region for the query subject, the corresponding value vectors are then weighted and summed based on cross-modal attention weights to generate a visual context vector containing information about the other modality. and text context vector : ; ; in, This is a normalization function used to calculate cross-modal attention weights; The dimension of the key vector; , They are respectively and The transpose of .
[0057] The two generated context vectors are fed back to the encoder of the corresponding modality via a residual connection to generate the visual and text vectors for the next layer. ; .
[0058] By stacking three layers of cross-modal fusion coding modules, multiple rounds of feature interaction and fusion are completed to obtain a visual representation that aggregates global semantic information. and text representation Extract from the two types of final representations respectively Vector sum Vectors are used as the final visual feature output. and final text feature output : ; ; in, , They are respectively and Dimensions.
[0059] The similarity calculation module is used to calculate image-text similarity, visual similarity, and text similarity.
[0060] Specifically, the final visual features will be output. and final text feature output Image feature vectors are obtained by mapping them to the same image-text joint semantic space using a projection matrix. and text feature vectors : ; ; in, , The learnable linear projection matrix is used to respectively... Vie and Vie Linear mapping to In the dimensional graph-text joint semantic space.
[0061] Will and Perform L2 normalization while preserving dimensions. Dimensional invariance: ; ; in, , These are the normalized visual feature vector and text feature vector, respectively.
[0062] Calculate the image-text similarity of the student model: ; in, The image-text similarity matrix for the student model; for The transpose of .
[0063] Calculate the visual and textual similarity of the student models: ; ; in, , These are the visual similarity matrix and text similarity matrix of the student model, respectively; for The transpose of .
[0064] Calculate the loss function and train the student model using mask distillation, such as Figure 3 As shown, it specifically includes: Calculate the distillation loss of teacher-student similarity: Image-text similarity based on teacher model A frozen teacher temperature parameter is introduced, and the similarity is converted into a probability distribution by row (or column) using the Softmax function. : ; in, Let be the matching probability of the i-th image sample corresponding to the j-th text sample in the teacher model; Let be the image-text similarity between the i-th image sample and the j-th text sample in the teacher model; For the Softmax function; For the frozen teacher temperature parameter, This is used to scale the image-text similarity of the teacher model.
[0065] Image-text similarity based on student model A learnable student temperature parameter is introduced, and the similarity is converted into a probability distribution by row (or column) using the Softmax function. : ; in, Let be the matching probability of the i-th image sample corresponding to the j-th text sample in the student model; Let i be the image-text similarity between the i-th image sample and the j-th text sample in the student model; For learnable student temperature parameters, It is used to scale the image-text similarity of the student model and is updated synchronously with the weights of the student model during training.
[0066] Through calculation and The KL divergence between them yields the teacher-student similarity distillation loss. : ; in, This indicates that the similarity is calculated along the rows of the similarity matrix (conditional on each image). and The sum of the KL divergences between them, where i is the index of the image sample; This indicates that the similarity is calculated along the column direction of the similarity matrix (conditional on each text). and The sum of KL divergences between the samples, where j is the index of the text sample.
[0067] Calculate the distillation loss of multimodal similarity between teachers and students : ; in, The loss term is the visual modality distillation term, which represents the visual similarity of the teacher model. Visual similarity with student models KL divergence between them; The text modal distillation loss term represents the text similarity of the teacher model. Text similarity with student model KL divergence between them; Since the visual modal distillation loss and the text modal distillation loss correspond to independent modal branches, they are accumulated with the same weight during the loss fusion process, and their respective weight coefficients are all 1.
[0068] Calculate the multimodal fusion similarity distillation loss: An adaptive weighting mechanism with multimodal gradient coordination is introduced to calculate the cross-entropy loss between the visual and textual similarity of the teacher model and the image-text similarity of the student model, respectively, to obtain the visual and textual gradients of the student model: ; ; in, Indicates the parameters of the student model; CE is the cross-entropy loss function; For visual distillation targets, calculate the visual similarity of the teacher model. Image-text similarity with student model Cross-entropy loss between; For text distillation, calculate the text similarity of the teacher model. Image-text similarity with student model Cross-entropy loss between them.
[0069] Calculate the gradients of the two distillation objectives with respect to the student model parameters: ; ; in, , These are the gradients of the visual distillation target and the text distillation target, respectively. This is the gradient operator.
[0070] By introducing visual weight coefficients and text weight coefficients, and minimizing the weighted gradient norm... Determine the optimal weight allocation, and use the optimal weights to perform a weighted fusion of the visual similarity and textual similarity of the teacher model to obtain the multimodal fused similarity: ; ; ; in, , These are the visual weight coefficient and the text weight coefficient, respectively. The square of the L2 norm; This represents the similarity score for multimodal fusion.
[0071] Calculate the distillation loss of multimodal fusion similarity based on the KL divergence between the image-text similarity of the multimodal fusion similarity and the similarity between the student model and the image-text similarity. : ; in, For multimodal fusion similarity Image-text similarity with student model The KL divergence between them.
[0072] A text-image comparison learning loss is constructed based on text-image matching and real labels to ensure that the feature output of the student model is consistent with the real labels, preventing the student model from learning incorrect knowledge from the teacher model. ; ; ; Where N is the batch size, i.e. the number of text-image pairs input into the model; In the student model, the Softmax function converts the similarity into a probability distribution corresponding to the index matching probability value, row by row. The loss of visual perception of text; In the student model, the Softmax function converts the similarity into a probability distribution corresponding to the index matching probability value. The loss in text-to-visual conversion; The learning loss is due to the comparison between text and images.
[0073] The total loss is obtained by weighted summation of the four types of losses mentioned above. : ; in, This is a hyperparameter.
[0074] S4, as Figure 4 As shown, high-quality image-text pairs using unmasked images are used in the dataset. The student model is fine-tuned by using only the image-text comparison learning loss as the optimization objective, in order to reduce the information loss caused by the mask and improve the model accuracy.
[0075] It should be noted that the present invention is not limited to the above-described embodiments. The above embodiments are merely examples, and any embodiments that have the same structure and perform the same effects as the technical concept within the scope of the present invention are included within the scope of the present invention. Furthermore, various modifications that can be conceived by those skilled in the art to the embodiments, and other ways of constructing by combining some of the constituent elements of the embodiments, without departing from the spirit of the present invention, are also included within the scope of the present invention.
Claims
1. A multimodal knowledge distillation method, characterized in that, include: S1. Obtain the image-text pair dataset and preprocess the image-text pairs using a pre-trained large model to obtain a high-quality image-text pair dataset. S2, input the high-quality image-text pair dataset into the pre-trained teacher multimodal model, and extract the image-text similarity knowledge, visual similarity knowledge and text similarity knowledge of the teacher model; S3, construct a lightweight student model, and use a high-quality image and text dataset to train the student model by mask distillation; S4 uses a high-quality unmasked image-text pair dataset and fine-tunes the student model using only the image-text contrast learning loss as the optimization objective to obtain the final student model.
2. The multimodal knowledge distillation method according to claim 1, characterized in that, S1 specifically includes: S11, use the image text description generation model to generate text for each original image in the image-text pair dataset, and together with the original text, form a candidate text set; S12 uses an image-text matching model to calculate the matching score between each original image and each text in the candidate text set, selects the text with the highest score as the optimal text description, and forms a high-quality image-text pair dataset with the original image.
3. The multimodal knowledge distillation method according to claim 1, characterized in that, S2 specifically includes: The image-text pairs are input into the visual encoder and text encoder of the teacher model respectively to obtain the original visual feature vector and the original text feature vector. After being projected into the joint semantic space and normalized, the image-text cosine similarity is calculated to obtain the image-text similarity knowledge of the teacher model. Visual similarity knowledge is obtained by calculating the cosine similarity between different images using normalized visual feature vectors. By calculating the cosine similarity between different texts using the normalized text feature vectors, we can obtain text similarity knowledge.
4. The multimodal knowledge distillation method according to claim 1, characterized in that, The student model includes: A visual encoder is used to extract visual features from an image; A text encoder is used to extract semantic features from text. A cross-modal fusion coding module is used to enable bidirectional interaction between visual and textual data; The similarity calculation module is used to calculate image-text similarity, visual similarity, and text similarity.
5. The multimodal knowledge distillation method according to claim 4, characterized in that, The visual encoder adopts a hierarchical semantic grouping visual encoding structure, which includes multiple encoding stages. In the encoding stage, learnable semantic aggregation vectors are introduced, and the current visual feature vector is gradually aggregated into the corresponding number of group vectors through grouping and aggregation operations.
6. The multimodal knowledge distillation method according to claim 5, characterized in that, The group aggregation operation includes: Calculate the correlation matrix between visual feature vectors and semantic aggregation vectors; Hard-valued semantic grouping relationships are generated based on the correlation matrix, so that each visual feature vector belongs to only one semantic aggregation vector; Based on the hard-value semantic grouping relationship, visual feature vectors belonging to the same semantic aggregation vector are weighted and aggregated to obtain the visual feature vector of the next layer.
7. The multimodal knowledge distillation method according to claim 4, characterized in that, The cross-modal fusion coding module is specifically used for: The visual feature vectors output by the visual encoder and the text feature vectors output by the text encoder are projected onto the query, key, and value spaces, respectively. The query vector of the visual feature vector is used to retrieve the key-value pairs of the text feature vector, and a visual-to-text context vector is generated. The query vector of the text feature vector is used to retrieve the key-value pairs of the visual feature vector, and a text-to-visual context vector is generated. The generated context vector is fed back to the encoder of the corresponding modality through residual connections for updating, and multiple layers of cross-modal fusion coding modules are stacked to obtain the final visual representation and the final text representation that aggregate global semantic information.
8. The multimodal knowledge distillation method according to claim 1, characterized in that, The method of training the student model using mask distillation on a high-quality image and text dataset specifically includes: Randomly mask the images in the high-quality image-text pair dataset to construct a masked image-text pair dataset, and input it into the student model to calculate the image-text similarity, visual similarity, and text similarity of the student model. Calculate the distillation loss of teacher-student similarity based on the image-text similarity of the teacher and student models; Calculate the teacher-student multimodal similarity distillation loss based on visual and textual similarity of teacher and student models; An adaptive weighting mechanism for multimodal gradient coordination is introduced to weight and fuse the visual similarity and text similarity of the teacher model to obtain a multimodal fusion similarity. The multimodal fusion similarity distillation loss is then calculated based on the image-text similarity of the student model. Construct a text-image comparison learning loss based on text-image matching and real-label; The weighted sum of the above four types of losses is used as the total loss for training the student model.
9. The multimodal knowledge distillation method according to claim 8, characterized in that, The random masking process for images in the high-quality image-text dataset specifically includes: The image is divided into multiple image blocks, a binary mask vector is constructed, and a preset proportion of image blocks are randomly set as the mask, while the remaining image blocks are retained for encoding, resulting in a masked image-text pair dataset.
10. A multimodal knowledge distillation method according to claim 8, characterized in that, The adaptive weighting mechanism for multimodal gradient coordination specifically includes: The cross-entropy loss between the visual similarity and text similarity of the teacher model and the image-text similarity of the student model is calculated separately to obtain the visual gradient and text gradient of the student model. We introduce visual weight coefficients and text weight coefficients, and determine the optimal weight allocation by minimizing the weighted gradient norm. The visual similarity and textual similarity of the teacher model are weighted and fused using the optimal weights to obtain the multimodal fused similarity.