Knowledge distillation method based on triple multi-mode prompt

Abstract

The invention relates to the technical field of machine learning, and provides a knowledge distillation method based on triple multi-mode prompt, which is characterized by comprising the following steps: acquiring a plurality of open source visual data sets and constructing a training set, a verification set and a test set; constructing a teacher model for distillation, and performing two-stage prompt fine tuning on the teacher model through training; constructing a student mode based on triple prompt distillation, and distilling the teacher model knowledge to a student model; and performing retrieval prediction on the trained student model. According to the method, through the prototype-guided triple prompt distillation mechanism and multi-loss collaborative optimization, efficient and accurate migration of cross-modal knowledge is realized, and the generalization performance and robustness of the model in a few-sample scene are comprehensively improved.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning, and in particular relates to a knowledge distillation method based on triple multimodal prompts. Background Art

[0002] With the breakthrough progress of large-scale pre-trained vision-language models (VLM) in multimodal tasks such as image classification, object detection, and image-text retrieval, downstream adaptation based on pre-trained models has become a research hotspot. Among them, prompt learning, with its advantages of efficient parameters and fast convergence, is widely used to transfer general models to specific domain tasks with few samples or no annotations.

[0003] Furthermore, knowledge distillation (KD) technology has become an important means of improving the performance of small models in lightweight deployment scenarios. The classic KD framework achieves soft-label knowledge transfer by minimizing the KL divergence between the output distributions of the teacher and student models. In recent years, methods such as PromptKD have combined prompt learning with distillation, leveraging the textual branch features of the teacher model to guide the training of the student's visual branch on unlabeled images, achieving considerable success.

[0004] However, existing distillation methods still suffer from the following shortcomings: neglect of negative cue information, lack of multiple contrast constraints, domain adaptation and catastrophic forgetting, and imbalance between training and inference efficiency. Specifically, methods such as PromptKD only use the positive text embedding of "This is a [category A]" as the distillation target, but fail to fully exploit the contrastive supervision signal contained in negative cues such as "This is not a [category A]", making it difficult to form sufficient discriminative ability at category boundaries. They only perform unary alignment between the teacher and student outputs, ignoring the triplet contrastive learning signal that simultaneously pulls image features closer to positive cues and further away from negative cues, resulting in insufficient model adaptability to hard examples. When distilling on low-sample or unlabeled domains, the teacher model has difficulty fully transferring knowledge from the source domain to the target domain, and the student model is prone to catastrophic forgetting, which reduces generalization performance to new categories. Although some hybrid cue or distillation methods have improved accuracy, the addition of a cue module or distillation loss significantly increases the training computational overhead. Inference still requires the use of additional cue parameters, which affects deployment efficiency. Summary of the Invention

[0005] In view of the above-mentioned shortcomings of the existing technology, the present invention proposes a knowledge distillation method based on triple multimodal prompts. This method can fully utilize positive and negative prompt information, introduce ternary contrast constraints in the distillation process, and take into account both training and inference efficiency. The technical solution designed by the present invention includes the following steps: S1: Obtain multiple open-source visual datasets and construct training sets, validation sets, and test sets; S2: Build a teacher model for distillation and perform two-stage prompt fine-tuning on the teacher model through training; S3: Construct a student model based on triplet prompt distillation and distill the teacher model knowledge into the student model; S4: Use the trained student model for retrieval prediction.

[0006] Preferably, the S2 includes: S21: Construct a few-shot prompt set based on the training set, and design positive prompt templates and negative prompt templates. S22: Input the few-shot prompt set into the visual encoder of the teacher model to obtain normalized visual features, input the positive prompt template and the negative prompt template into the text encoder of the teacher model to obtain positive text features and negative text features, fine-tune the text prompt vector and the visual encoder based on the cross-entropy loss, and solidify the positive text features and negative text features after fine-tuning converges.

[0007] Preferably, the normalized visual features are obtained in S22 using the following formula:

[0008] Where, is the normalized visual feature, is the visual encoder of the teacher model, is a sample prompt set.

[0009] Preferably, the formula for obtaining the positive example text features and the negative example text features in S22 is as follows:

[0010]

[0011] Where, is the positive text feature, is the negative text feature, and They are and The i-th and j-th samples in , is the text encoder of the teacher model, and They are positive prompt template and negative prompt template respectively. is the number of categories.

[0012] Preferably, the cross entropy loss in S22 is calculated as follows:

[0013] Where, is the cross entropy loss, is the true label of the sample, is the temperature coefficient.

[0014] Preferably, the S3 includes: S31: On the unlabeled domain dataset, each input image The fine-tuned teacher model visual encoder and the student model visual encoder to be trained are respectively fed into the forward inference and normalized to obtain the teacher image features. and student image features ; S32: Student image features are transformed through the learnable mapping layer Align to teacher image features Dimensions; S33: Student image features and teacher image features Perform dot product operations with the saved positive text features of the teacher model to obtain the logical distribution prediction of the teacher model and the logistic distribution predictions of the Student model ; S34: Distilling the teacher model’s logistic distribution predictions and the logistic distribution predictions of the Student model alignment; S35: Optimize the student model visual encoder based on the combined loss function until the distillation training is completed.

[0015] Preferably, the teacher image features are obtained in S31 and student image features , the formula is as follows:

[0016]

[0017] Where, is the visual encoder of the teacher model, The visual encoder for the student model.

[0018] Preferably, the learnable mapping layer in S32 has the following formula:

[0019] Where, is a learnable mapping layer, is the convolution operation, is a nonlinear activation function, It is batch normalization.

[0020] Preferably, the logical distribution prediction of the teacher model is obtained in S33 and the logistic distribution predictions of the Student model , the formula is as follows:

[0021]

[0022] Where, It is the saved positive text features of the teacher model.

[0023] Preferably, the combined loss function in S35 is as follows:

[0024]

[0025]

[0026]

[0027] Where, is the combined loss function, 、 and They are feature alignment loss, knowledge distillation loss and triple loss respectively. is the Softmax function, is the temperature coefficient, is the Kullback-Leibler divergence operation, is the distance calculation function, is a hard positive sample, is a hard negative sample, 、 、 and is a hyperparameter and B is the batch size.

[0028] Beneficial effects: This application provides a knowledge distillation method based on triplet multimodal prompts. Through the prototype-guided triplet prompt distillation mechanism and multi-loss collaborative optimization, it realizes the efficient and accurate transfer of cross-modal knowledge, and comprehensively improves the generalization performance and robustness of the model in few-sample scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1It is a flow chart of a preferred embodiment of the present invention; Figure 2 It is a schematic diagram of the structure of a multimodal unsupervised prompt learning model of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0030] The embodiments of the present invention are described in detail below. The following embodiments are implemented based on the technical solutions of the present invention, and provide detailed implementation methods and specific operating procedures. However, the protection scope of the present invention is not limited to the following embodiments.

[0031] The present invention designs a knowledge distillation method based on triple multimodal prompts, and the technical solution includes the following steps: Figure 1-2 As shown, specifically including: S1: Obtain multiple open-source visual datasets and construct training sets, validation sets, and test sets; S2: Build a teacher model for distillation and perform two-stage prompt fine-tuning on the teacher model through training; S3: Construct a student model based on triplet prompt distillation and distill the teacher model knowledge into the student model; S4: Use the trained student model for retrieval prediction.

[0032] Specifically, for S1, we first collected 11 public large-scale visual datasets. After preprocessing, each dataset was divided into a base subset and a novel subset by category. Then, based on the triplet prompt distillation (PTPD) framework, we constructed a multimodal model and trained and optimized it using the training data on the base and novel subsets. After training, the classification performance of the model was evaluated on the validation and test sets. Finally, the trained PTPD model was applied to inference of online images to verify its generalization ability in real-world scenarios. Among them, the original data and annotations of the 11 public large-scale visual datasets were obtained from their official websites or public repositories in turn: ImageNet (ILSVRC2012) dataset, download link: https: / / www.image-net.org / download.php ; Caltech101 dataset, download link: http: / / www.vision.caltech.edu / Image_Datasets / Caltech101 / Oxford Pets dataset, download link: https: / / www.robots.ox.ac.uk / ~vgg / data / pets / ; Stanford Cars dataset, download link: https: / / huggingface.co / datasets / tanganke / stanford_cars Flowers102 dataset, download link: https: / / www.robots.ox.ac.uk / ~vgg / data / flowers / 102 / ; Food-101 dataset, download link: https: / / data.vision.ee.ethz.ch / cvl / datasets_extra / food-101 / ; FGVC Aircraft dataset, download link: https: / / www.robots.ox.ac.uk / ~vgg / data / fgvc-aircraft / ; SUN397 dataset, download link: https: / / vision.princeton.edu / projects / 2010 / SUN / ; DTD (DescribableTextures Dataset) dataset, download link: https: / / www.robots.ox.ac.uk / ~vgg / data / dtd / ; EuroSAT dataset, download link: http: / / madm.dfki.de / downloads ; UCF101 dataset, download link: https: / / www.crcv.ucf.edu / data-sets / ucf101 / We then cleaned the dataset and unified the import formats of different datasets. When loading the preprocessed dataset, we first traverse the index file in the unified format, read each image and its corresponding label number by path, and then convert the image into a tensor format acceptable to the model through a predefined data augmentation and normalization pipeline.

[0033] Furthermore, to simulate few-shot learning scenarios, during the prompt fine-tuning phase of the teacher model, we randomly sampled a fixed set of 16 images from each category to construct a "few-shot training set." This ensures that representative examples from each category participate in prompt learning while also controlling the data size and avoiding data redundancy. The sampling process is seeded with a random number to ensure experimental reproducibility. During the unsupervised distillation phase of the student model, we use the full set of examples, including both the base and novel classes, to expose the model to the widest range of categories and improve generalization to unseen categories. To monitor model performance, prevent overfitting, and optimize hyperparameters during training, we split the original full training set into two equal parts: 1. Validation Set: This uses the first 50% of samples to evaluate the student model's performance on the target domain, such as classification accuracy and recall, after each training cycle. This allows for dynamic adjustment of hyperparameters like the learning rate and weight decay. 2. Test Set: This retains the last 50% of samples as final evaluation data after training, objectively measuring the model's performance in real-world scenarios and ensuring the fairness and comparability of the results. The entire loading and partitioning process utilizes PyTorch's Dataset and DataLoader interfaces for efficient batch reading and multi-process prefetching. Before training, we also perform a secondary validation of the image paths, labels, and subset flags (Few-Shot, Zero-Shot, Validation, and Test) recorded in the index file to prevent training interruptions or evaluation bias caused by missing or incorrectly formatted files. This systematic data loading and partitioning strategy not only provides high-quality data for the teacher model's few-shot cue learning and the student model's zero-shot distillation, but also ensures the scientific nature and rigor of model tuning and final performance evaluation through a separate evaluation mechanism for validation and test sets.

[0034] Preferably, S2 includes: S21: Construct a few-shot prompt set based on the training set, and design positive prompt templates and negative prompt templates. S22: Input the few-shot prompt set into the visual encoder of the teacher model to obtain normalized visual features, input the positive prompt template and the negative prompt template into the text encoder of the teacher model to obtain positive text features and negative text features, fine-tune the text prompt vector and the visual encoder based on the cross-entropy loss, and solidify the positive text features and negative text features after fine-tuning converges.

[0035] Preferably, the normalized visual features are obtained in S22 using the following formula:

[0036] Where, is the normalized visual feature, is the visual encoder of the teacher model, is a sample prompt set.

[0037] Preferably, the formula for obtaining the positive example text features and the negative example text features in S22 is as follows:

[0038]

[0039] Where, is the positive text feature, is the negative text feature, and They are and The i-th and j-th samples in , is the text encoder of the teacher model, and They are positive prompt template and negative prompt template respectively. is the number of categories.

[0040] Preferably, the cross entropy loss in S22 is as follows:

[0041] Where, is the cross entropy loss, is the true label of the sample, is the temperature coefficient.

[0042] Specifically, for S2, according to the "Teacher Model (Few-Shot) Training Set Construction" in S1, 16 images are randomly sampled from the training set for each category, and the random seed is fixed to ensure reproducibility. The positive template T⁺ is: "This is a [category name] photo."; the negative template T⁻ is: "This is NOT a [category name] photo."; under the same random seed, a set of positive and negative prompt sentences are generated for each category to ensure that the positive and negative samples are balanced and the experiment is reproducible.

[0043] In addition, after the fine-tuning phase converges, the positive and negative text feature sets and Fixed, no need to update, store all N categories of teacher model positive text features as and negative text features , and these two sets of shared class vectors are used in the subsequent student model distillation stage.

[0044] Preferably, S3 includes: S31: On the unlabeled domain dataset, each input image The fine-tuned teacher model visual encoder and the student model visual encoder to be trained are respectively fed into the forward inference and normalized to obtain the teacher image features. and student image features ; S32: Student image features are transformed through the learnable mapping layer Align to teacher image features Dimensions; S33: Student image features and teacher image features Perform dot product operations with the saved positive text features of the teacher model to obtain the logical distribution prediction of the teacher model and the logistic distribution predictions of the Student model ; S34: Distilling the teacher model’s logistic distribution predictions and the logistic distribution predictions of the Student model Alignment; S35: Optimize the student model visual encoder based on the combined loss function until the distillation training is completed.

[0045] Preferably, the teacher image features are obtained in S31 and student image features , the formula is as follows:

[0046]

[0047] Where, is the visual encoder of the teacher model, The visual encoder for the student model.

[0048] Preferably, the learnable mapping layer in S32 has the following formula:

[0049] Where, is a learnable mapping layer, is the convolution operation, is a nonlinear activation function, It is batch normalization.

[0050] Preferably, the logical distribution prediction of the teacher model is obtained in S33 and the logistic distribution predictions of the Student model , the formula is as follows:

[0051]

[0052] Where, It is the saved positive text features of the teacher model.

[0053] Preferably, the combined loss function in S35 is as follows:

[0054]

[0055]

[0056]

[0057] Where, is the combined loss function, 、 and They are feature alignment loss, knowledge distillation loss and triple loss respectively. is the Softmax function, is the temperature coefficient, is the Kullback-Leibler divergence operation, is the distance calculation function, is a hard positive sample, is a hard negative sample, 、 、 and is a hyperparameter and B is the batch size.

[0058] Specifically, for S3, this stage constructs a triplet prompt learning task, and uses the positive and negative prompt features of the teacher model as a reference to train the student model. By minimizing the gap between the student output and the teacher output, the student model is prompted to produce representations and prediction results that are highly consistent with the teacher model under the same prompts. Given that the teacher model is much larger than the student model, we added a learnable mapping layer to the visual encoder of the student model to map the student features to the same dimension as the teacher features with lower computational overhead, thereby more efficiently achieving feature alignment between the two, and aligning the predicted distribution of the student model with the predicted distribution of the teacher model through distillation loss (for example, minimizing the KL divergence between the two) to optimize the parameters of the student image encoder. After the distillation training is completed, the obtained student visual encoder It can be combined with the previously saved positive text feature set , as a lightweight model for inference in subsequent downstream tasks.

[0059] Furthermore, in the design of the combined loss function, the knowledge distillation loss and the image feature alignment loss work together to transfer the soft label information and high-level semantic features learned by the teacher model during large-scale pre-training to the lightweight student model, maximizing the preservation of key information while significantly reducing the computational burden of the student model. The boundary-constrained triplet loss leverages the positive and negative prototypes constructed by the teacher model on a small amount of annotated domain data to enhance the student model's discriminative ability during unsupervised training, thereby improving its generalization performance in the target domain.

[0060] In the positive sample set, the sample with the largest embedding distance from the anchor image is selected as (hard positive sample); in the negative sample set, the sample with the smallest distance to the anchor point is selected as (Hard negative samples), the triplet loss is calculated based on the hard positive samples, hard negative samples and student image features, and margin is a hyperparameter used to control the minimum distance between the positive and negative sample prototypes and the image samples.

[0061] Through knowledge distillation, the image encoder output features of the pre-trained teacher model are aligned with the output features of the student model using KL divergence and L1 loss, respectively. A triplet loss with boundary constraints is also introduced to ensure accurate and efficient knowledge transfer. This staged architecture not only optimizes the organization of the training process but also improves overall training efficiency, ultimately outputting a combined loss function for the model.

[0062] In addition, the experimental setup of this application includes: The teacher model was trained with a hint depth of 9 and a hint length of 4 for both visual and text hints. Stochastic gradient descent (SGD) was used as the optimizer. All student models were trained for 20 epochs with a batch size of 8 and a learning rate of 0.005. The coefficients in the loss function were 、 They are 10 and 1 respectively. When the data set is DTD, FGVCAircraft, Flowers102, is 200 for the first dataset and 1000 for the other datasets. The temperature hyperparameter τ in the distillation method is set to 1.0; The Base class and Novel class are used as evaluation criteria, and the accuracy of the Base class and Novel class are comprehensively considered, and the harmonic mean (HM) of the accuracy on the Base class and Novel class is used as the evaluation indicator.

[0063] The above describes in detail the preferred embodiments of the present invention. It should be understood that numerous modifications and variations based on the concepts of the present invention are possible by those skilled in the art without inventive effort. Therefore, any technical solution that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A knowledge distillation method based on triple multimodal prompts, characterized in that: include: S1: Obtain multiple open-source visual datasets and construct training sets, validation sets, and test sets; S2: Build a teacher model for distillation and perform two-stage prompt fine-tuning on the teacher model through training; S3: Construct a student model based on triplet prompt distillation and distill the teacher model knowledge into the student model; S4: Use the trained student model for retrieval prediction.

2. A knowledge distillation method based on triple multimodal prompts according to claim 1, characterized in that: The S2 includes: S21: Construct a few-shot prompt set based on the training set, and design positive prompt templates and negative prompt templates. S22: Input the few-shot prompt set into the visual encoder of the teacher model to obtain normalized visual features, input the positive prompt template and the negative prompt template into the text encoder of the teacher model to obtain positive text features and negative text features, fine-tune the text prompt vector and the visual encoder based on the cross-entropy loss, and solidify the positive text features and negative text features after fine-tuning converges.

3. A knowledge distillation method based on triple multimodal prompts according to claim 2, characterized in that: The normalized visual features obtained in S22 are formulated as follows: Where, is the normalized visual feature, is the visual encoder of the teacher model, is a sample prompt set.

4. A knowledge distillation method based on triple multimodal prompts according to claim 3, characterized in that: The formula for obtaining the positive example text features and the negative example text features in S22 is as follows: Where, is the positive text feature, is the negative text feature, and They are and The i-th and j-th samples in , is the text encoder of the teacher model, and They are positive prompt template and negative prompt template respectively. is the number of categories.

5. The knowledge distillation method based on triple multimodal prompts according to claim 4 is characterized in that: The cross entropy loss in S22 is as follows: Where, is the cross entropy loss, is the true label of the sample, is the temperature coefficient.

6. The knowledge distillation method based on triple multimodal prompts according to claim 1, characterized in that: The S3 includes: S31: On the unlabeled domain dataset, each input image The fine-tuned teacher model visual encoder and the student model visual encoder to be trained are respectively fed into the forward inference and normalized to obtain the teacher image features. and student image features ; S32: Student image features are transformed through a learnable mapping layer Align to teacher image features Dimensions; S33: Student image features and teacher image features Perform dot product operations with the saved positive text features of the teacher model to obtain the logical distribution prediction of the teacher model and the logistic distribution predictions of the Student model ; S34: Distilling the teacher model’s logistic distribution predictions and the logistic distribution predictions of the Student model Alignment; S35: Optimize the student model visual encoder based on the combined loss function until the distillation training is completed.

7. The knowledge distillation method based on triple multimodal prompts according to claim 6, characterized in that: The teacher image features are obtained in S31 and student image features , the formula is as follows: Where, is the visual encoder of the teacher model, The visual encoder for the student model.

8. The knowledge distillation method based on triple multimodal prompts according to claim 7, characterized in that: The learnable mapping layer in S32 has the following formula: Where, is a learnable mapping layer, is the convolution operation, is a nonlinear activation function, It is batch normalization.

9. The knowledge distillation method based on triple multimodal prompts according to claim 8, characterized in that: The logical distribution prediction of the teacher model is obtained in S33 and the logistic distribution predictions of the Student model , the formula is as follows: Where, is the saved positive text features of the teacher model.

10. A knowledge distillation method based on triple multimodal prompts according to claim 9, characterized in that: The combined loss function in S35 is as follows: Where, is the combined loss function, 、 and They are feature alignment loss, knowledge distillation loss and triple loss respectively. is the Softmax function, is the temperature coefficient, is the Kullback-Leibler divergence operation, is the distance calculation function, is a hard positive sample, is a hard negative sample, 、 、 and is a hyperparameter and B is the batch size.

Images