Multimodal prompting fine-tuning method, fine-tuning system, device and medium
By using a multimodal prompting fine-tuning method, core semantic features are learned and the network model is optimized, which solves the problems of feature drift and forgetting in persistent image classification with few samples, and improves the recognition performance and stability of the model under few sample conditions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CIVIL AVIATION MANAGEMENT INSTITUTE OF CHINA
- Filing Date
- 2025-11-19
- Publication Date
- 2026-07-21
AI Technical Summary
When performing continuous image classification with few samples under limited sample conditions, existing technologies are prone to problems such as feature drift, forgetting of old classes, or insufficient learning of new classes, making it difficult to achieve a balance between maintaining the recognition performance of old classes and quickly adapting to new classes.
We employ a multimodal cue fine-tuning method. By learning the core semantic features of each category, we design category-specific cue vectors to guide the extraction of representations that are more relevant to category knowledge from both text and image modalities. We use cross-entropy loss and KL divergence loss to optimize the network model, enabling selective activation and fusion updates of old knowledge.
It improves the model's recognition performance and stability under conditions with few samples, reduces the forgetting rate in image classification, and enhances the model's adaptability in environments with sparse samples and continuous tasks.
Smart Images
Figure CN121544936B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image classification technology, and in particular to a multimodal cue fine-tuning method, fine-tuning system, device and medium. Background Technology
[0002] Few-Shot Class-Incremental Learning (FSCIL) is an important machine learning task that aims to retain existing knowledge and correctly identify new classes while gradually introducing new class data to the model, given an extremely limited number of samples per class. In real-world applications, data exhibits dynamic growth, requiring models to continuously incorporate new class information without relying on large-scale samples. This process is affected by factors such as catastrophic forgetting, class imbalance, and sample scarcity, making it a significant research direction in the field of continuous learning in recent years. With the widespread application of artificial intelligence in areas such as security monitoring, intelligent manufacturing, and human-computer interaction, few-shot class-incremental continuous learning is crucial for ensuring system stability, adaptability, and long-term performance.
[0003] Compared to traditional persistent image classification, few-shot persistent image classification requires rapid learning of new classes under extremely low sample conditions. Therefore, it is particularly important in scenarios where data collection is difficult, annotation costs are high, or privacy is restricted (such as medical image analysis and industrial anomaly detection). However, when faced with new classes, models often need to strike a balance between maintaining the recognition performance of old classes and rapidly adapting to new classes. The extremely small sample size of newly added classes further increases the learning difficulty, leading to performance degradation and recognition bias. This challenge is particularly prominent in real-world systems; for example, intelligent security systems need to continuously identify new target classes, but the significant differences in data volume between different classes make traditional methods unsuitable. These needs have driven the rapid development of research on few-shot incremental persistent learning to improve the generalization ability and long-term adaptability of models.
[0004] Current methods primarily improve model stability during the incremental phase through knowledge distillation, feature regularization, memory replay, and meta-learning. However, when samples are extremely scarce, these methods are still prone to problems such as feature drift, forgetting of old classes, or insufficient learning of new classes. Furthermore, meta-learning methods rely on task construction and struggle to cope with the complexity of class distributions in real-world scenarios; memory replay methods, on the other hand, are limited by storage capacity and cannot fully retain historical information. Summary of the Invention
[0005] To address the aforementioned shortcomings, this application proposes a multimodal cue fine-tuning method, fine-tuning system, device, and medium. By learning the core semantic features of each category, the robustness of the features is enhanced, and the recognition performance and stability of the model in sparse sample and continuous task environments are improved without the use of replay.
[0006] This application discloses a multimodal cue fine-tuning method, which includes: Step 1: Obtain the dataset and divide it into a training dataset and a test dataset. The training dataset is further divided into a first training dataset and a second training dataset. Both the first and second training datasets consist of image samples and their corresponding category labels. The category labels indicate the category to which the image samples belong. The number of image samples corresponding to each category in the second training dataset is less than the number of image samples corresponding to each category in the first training dataset. Step 2: Construct the network model, which includes a CLIP pre-trained text encoder, a CLIP pre-trained visual encoder, a cue vector selection network, and a projection network; Step 3: Input the first training dataset into the network model for training to obtain the first network model; input the second training dataset into the first network model for training to obtain the second network model; Step 4: Input the images in the test dataset into the second network model to obtain the image classification results, which are the categories to which the images belong.
[0007] Furthermore, the CLIP pre-trained text encoder adopts a Transformer-based structure; the CLIP pre-trained visual encoder adopts a visual Transformer structure; the cue vector selection network is a multilayer perceptron, which consists of two linear layers and an activation layer located between them; the projection network is a feedforward neural network, which consists of linear layers.
[0008] Further, the step of inputting the first training dataset into the network model for training to obtain the first network model includes: Step 31: After segmenting all category labels in the first training dataset, input them into the CLIP pre-trained text encoder in the network model to obtain the text representation of each category label; Step 32: Input the text representation of each category label into the cue vector selection network in the network model to obtain a probability score; the probability score represents the matching degree between each cue vector in the cue vector pool and each category label; the cue vector is used to guide the network model to obtain the expected image classification result; Step 33: Based on the probability scores, obtain the category-specific cue vector corresponding to each category label; based on each category-specific cue vector and the CLIP pre-trained text encoder, obtain the text features of each category label in the first training dataset; Step 34: Input all image samples from the first training dataset into the CLIP pre-trained visual encoder in the network model to obtain the image representation of each image sample; Step 35: Obtain the visual features of each image sample based on the image representation of each image sample, the text representation of all the category labels, and the corresponding category-specific cue vectors; Step 36: Calculate the similarity between the visual features of each image sample and the text features of each current category label, and assign the category label corresponding to the maximum similarity; the category label is used to indicate the category to which each image sample belongs; Step 37: After executing steps 31 to 36, output the trained network model, i.e., the first network model.
[0009] Further, step 33 includes: Step 331: Select the top k cue vectors with the highest probability scores, and use the top k cue vectors as the category-specific cue vectors corresponding to the category label; the category-specific cue vectors are the core semantic features most relevant to the category corresponding to the first training dataset; Step 332: Concatenate the category-specific prompt vector and its corresponding category name to obtain the text input vector; Step 333: Input the text input vector into the multi-layer Transformer encoder of the CLIP pre-trained text encoder to obtain the enhanced text representation, i.e., text features.
[0010] Further, step 35 includes: Step 351: Calculate the similarity between the image representation of each image sample and the text representation of all the category labels to obtain multiple similarity scores; Step 352: Map the category-specific cue vectors of all the category labels to the visual space through the projection network in the network model to form intermediate visual cue vectors; Step 353: Use the multiple similarities as weights to weight the intermediate visual cue vectors in the intermediate visual cue vectors to obtain the final visual cue vector for each image sample; Step 354: Concatenate the final visual cue vector, category terminology, and image patch corresponding to each image sample, and input the concatenated vector into the CLIP pre-trained visual encoder to obtain the enhanced image representation, i.e., visual features; the category terminology is used to summarize the global features of each image sample.
[0011] Furthermore, during the training of the network model and the first network model, cross-entropy loss is used to optimize the trainable parameters; the trainable parameters include all cue vectors in the cue vector pool, cue vector selection network parameters, and projection network parameters. The trainable parameters are optimized using the backpropagation algorithm.
[0012] Furthermore, the expression for the cross-entropy loss is:
[0013]
[0014] in, For cross-entropy loss, The first prediction of the network model or the first network model The category labels of each image sample For the first The number of categories for all image samples included in an image classification task. For the first The true class label corresponding to each image sample It is a binary function, when hour, The value of is 1, when hour, The value of is 0. For the network model based on image samples The predicted category label is The probability, It is a logarithmic function. It is an exponential function. For the first Text features and the visual features The cosine value between Represents text, Represents vision, This is the temperature coefficient hyperparameter. The number of categories for each image sample, The total number of classes in all image samples in the training dataset. This is the summation symbol.
[0015] Furthermore, during the training of the cue vector selection network, KL divergence loss is used to ensure that the cue vector selection behavior between categories is consistent with their semantic similarity. This means that categories with greater similarity have a higher probability of selecting the same cue vector, thereby providing similar knowledge for similar categories and improving the ability of the network model and the first network model to adapt to new image classification tasks. The knowledge refers to the core semantic features most relevant to each category.
[0016] Furthermore, the process of obtaining the KL divergence loss includes: Calculate the similarity between the text representations of each category label in the first training dataset and the second training dataset, and construct a similarity matrix from all the obtained similarities; Each category label from the first training dataset and the second training dataset is input into the prompt vector selection network to obtain the corresponding probability score; the similarity between the probability scores corresponding to each category label is calculated, and all the obtained similarities are used to form a distribution similarity matrix. The KL divergence loss is determined based on the similarity matrix and the distribution similarity matrix.
[0017] Further, the KL divergence loss is determined based on the similarity matrix and the distribution similarity matrix, including: The KL divergence loss is obtained using the following formula:
[0018] in, For KL divergence loss, For category Corresponding similarity matrix Similarity matrix with distribution The divergence loss between them Distance metric symbol, For category-based Accumulate. It is a logarithmic function.
[0019] This application also discloses a multimodal cue fine-tuning system that implements the aforementioned multimodal cue fine-tuning method, comprising: A dataset acquisition module is used to acquire a dataset, divide the dataset into a training dataset and a test dataset, and divide the training dataset into a first training dataset and a second training dataset; both the first training dataset and the second training dataset consist of image samples and their corresponding category labels; the category labels are used to indicate the category to which the image samples belong; the number of image samples corresponding to each category in the second training dataset is less than the number of image samples corresponding to each category in the first training dataset; The model building module is used to build a network model, which includes a CLIP pre-trained text encoder, a CLIP pre-trained visual encoder, a cue vector selection network, and a projection network. The model training module is used to input the first training dataset into the network model for training to obtain a first network model; and to input the second training dataset into the first network model for training to obtain a second network model. The image classification module is used to input the images in the test dataset into the second network model to obtain the image classification result, which is the category to which the image belongs.
[0020] This application also discloses an electronic device, including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the method described above.
[0021] This application also discloses a computer-readable storage medium comprising a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described above.
[0022] Due to the adoption of the above technical solution, this application has the following advantages: 1. This application guides the extraction of representations more relevant to category knowledge by designing category-specific cue vectors for both text and image modalities, thereby achieving selective activation and fusion update of old knowledge; it improves the accuracy of image classification by learning core semantic features; and it reduces the forgetting rate of image classification by selectively updating core semantic features.
[0023] 2. The cue vector mechanism for category semantic relations in this application can selectively reuse and update existing knowledge when learning new categories, thereby mitigating the forgetting problem and improving the model's adaptability under conditions with few samples. Attached Figure Description
[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.
[0025] Figure 1 This is a flowchart illustrating a multimodal cue fine-tuning method according to an embodiment of this application; Figure 2 This is a flowchart illustrating another multimodal cue fine-tuning method according to an embodiment of this application; Figure 3This is a block diagram of a multimodal prompting fine-tuning system according to an embodiment of this application; Figure 4 This is a block diagram of an electronic device according to an embodiment of this application. Detailed Implementation
[0026] The present application will be further described in conjunction with the accompanying drawings and embodiments. The described embodiments are only some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art should fall within the protection scope of the present application.
[0027] See Figure 1 and Figure 2 This application provides an embodiment of a multimodal cue fine-tuning method, which includes: S1: Obtain the dataset and divide it into a training dataset and a test dataset. The training dataset is further divided into a first training dataset and a second training dataset. Both the first and second training datasets consist of image samples and their corresponding class labels. The class labels are used to indicate the class to which the image samples belong. The number of image samples corresponding to each class in the second training dataset is less than the number of image samples corresponding to each class in the first training dataset. S2: Construct the network model, which includes a CLIP pre-trained text encoder, a CLIP pre-trained visual encoder, a cue vector selection network, and a projection network; S3: Input the first training dataset into the network model for training to obtain the first network model; input the second training dataset into the first network model for training to obtain the second network model; S4: Input the images from the test dataset into the second network model to obtain the image classification results, which are the categories to which the images belong.
[0028] In one embodiment of this application, the CLIP pre-trained text encoder adopts a Transformer-based structure; the CLIP pre-trained visual encoder adopts a visual Transformer structure; the cue vector selection network is a multilayer perceptron (MLP), which consists of two linear layers and an activation layer located between them; the projection network is a feedforward neural network (FNN), which consists of linear layers.
[0029] In one embodiment of this application, a first training dataset is input into a network model for training to obtain a first network model, including: S31: After segmenting all category labels in the first training dataset, input them into the CLIP pre-trained text encoder in the network model to obtain the text representation of each category label; S32: Input the text representation of each category label into the cue vector selection network in the network model to obtain a probability score; the probability score represents the matching degree between each cue vector in the cue vector pool and each category label; the cue vector is used to guide the network model to obtain the expected image classification result; S33: Based on the probability scores, obtain the category-specific cue vector corresponding to each category label; based on each category-specific cue vector and the CLIP pre-trained text encoder, obtain the text features of each category label in the first training dataset; S34: Input all image samples from the first training dataset into the CLIP pre-trained visual encoder in the network model to obtain the image representation of each image sample; Step 35: Based on the image representation of each image sample, the text representation of all category labels, and the corresponding category-specific cue vector, obtain the visual features of each image sample; S36: Calculate the similarity between the visual features of each image sample and the text features of each current category label, and assign the category label corresponding to the maximum similarity; the category label is used to indicate the category to which each image sample belongs; S37: After executing S31 to S36, output the trained network model, i.e., the first network model.
[0030] Optionally, the first network model is tested using a first test dataset to obtain image classification results, and the accuracy of image classification is evaluated using these results.
[0031] In one embodiment of this application, S33 includes: S331: Select the top k cue vectors with the highest probability scores and use them as category-specific cue vectors corresponding to the category labels. The category-specific cue vectors are the core semantic features most relevant to the category corresponding to the first training dataset. The core semantic features represent the key information of the category label. The key information is the focus of judging the category label. For example, if the category label belongs to animals, the key information may be information such as hair, limbs, etc.; k is a positive integer. S332: Concatenate the category-specific cue vector and its corresponding category name to obtain the text input vector; S333: Input the text input vector into the multi-layer Transformer encoder of the CLIP pre-trained text encoder to obtain the enhanced text representation, i.e., text features.
[0032] In one embodiment of this application, S35 includes: S351: Calculate the similarity between the image representation of each image sample and the text representation of all category labels to obtain multiple first similarity scores; S352: Map the category-specific cue vectors of all category labels to the visual space through the projection network in the network model to form intermediate visual cue vectors; S353: Use multiple first similarities as weights to weight the intermediate visual cue vectors in the intermediate visual cue vectors to obtain the final visual cue vector for each image sample; S354: The final visual cue vector, category terms, and image patch corresponding to each image sample are concatenated and then input into the CLIP pre-trained visual encoder to obtain the enhanced image representation, i.e., visual features; the category terms are used to summarize the global features of each image sample.
[0033] In one embodiment of this application, the process of inputting a second training dataset into a first network model for training to obtain a second network model is as follows: Based on the second training dataset and the first network model, the trained network model, i.e., the second network model, can be obtained by following the steps S31 to S36 above. For example, the second training dataset can replace the first training dataset in the aforementioned steps (e.g., S31, S33), and the first network model can replace the network model in the aforementioned steps (e.g., S31, S32).
[0034] Optionally, the second network model can be tested using a second test dataset to obtain image classification results, and the accuracy of image classification can be evaluated using these results.
[0035] The second network model is tested using the first test dataset to obtain image classification results, and the accuracy of image classification is evaluated based on these results. The image classification accuracy of the first network model tested with the first test dataset is subtracted from the image classification accuracy of the second network model tested with the first test dataset to evaluate the forgetting rate of image classification.
[0036] In one embodiment of this application, during the training of the network model and the first network model, cross-entropy loss is used to optimize the trainable parameters; the trainable parameters include all cue vectors in the cue vector pool, cue vector selection network parameters, and projection network parameters; The trainable parameters are optimized using the backpropagation algorithm.
[0037] In one embodiment of this application, the expression for cross-entropy loss is:
[0038]
[0039] in, For cross-entropy loss, The first prediction of the network model or the first network model The category labels of each image sample For the first The number of categories for all image samples included in an image classification task. For the first The true class label corresponding to each image sample It is a binary function, when hour, The value of is 1, when hour, The value of is 0. For the network model based on image samples The predicted category label is The probability, It is a logarithmic function. It is an exponential function. For the first Text features and the visual features The cosine value between Represents text, Represents vision, This is the temperature coefficient hyperparameter. The number of categories for each image sample, The total number of classes in all image samples in the training dataset. This is the summation symbol.
[0040] In one embodiment of this application, during the training of the cue vector selection network, KL divergence loss is used to ensure that the cue vector selection behavior between categories is consistent with their semantic similarity. This means that categories with greater similarity have a higher probability of selecting the same cue vector, thereby providing similar knowledge for similar categories and improving the ability of the network model and the first network model to adapt to new image classification tasks. The knowledge is the core semantic features most relevant to each category.
[0041] In one embodiment of this application, the process of obtaining the KL divergence loss includes: Calculate the similarity between the text representations of each category label in the first training dataset or the second training dataset, and construct a similarity matrix from all the obtained similarities; Input each category label from the first or second training dataset into the prompt vector selection network to obtain the corresponding probability score; calculate the similarity between the probability scores corresponding to each category label, and construct a distribution similarity matrix from all the obtained similarities. The KL divergence loss is determined based on the similarity matrix and the distribution similarity matrix.
[0042] In one embodiment of this application, determining the KL divergence loss based on the similarity matrix and the distribution similarity matrix includes: The KL divergence loss is obtained using the following formula:
[0043] in, For KL divergence loss, For category Corresponding similarity matrix Similarity matrix with distribution The divergence loss between them Distance metric symbol, For category-based Accumulate. It is a logarithmic function.
[0044] See Figure 3 This application also provides an embodiment of a multimodal cue fine-tuning system, implementing the multimodal cue fine-tuning method described in the above embodiments, comprising: A dataset acquisition module is used to acquire a dataset, divide the dataset into a training dataset and a test dataset, and divide the training dataset into a first training dataset and a second training dataset; both the first training dataset and the second training dataset consist of image samples and their corresponding category labels; the category labels are used to indicate the category to which the image samples belong; the number of image samples corresponding to each category in the second training dataset is less than the number of image samples corresponding to each category in the first training dataset; The model building module is used to build a network model, which includes a CLIP pre-trained text encoder, a CLIP pre-trained visual encoder, a cue vector selection network, and a projection network. The model training module is used to input the first training dataset into the network model for training to obtain a first network model; and to input the second training dataset into the first network model for training to obtain a second network model. The image classification module is used to input the images in the test dataset into the second network model to obtain the image classification result, which is the category to which the image belongs.
[0045] See Figure 4This application also provides an electronic device including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, it implements the multimodal cue fine-tuning method described in the above embodiments. As an example, the electronic device may include multiple processors. A processor may refer to one or more devices, circuits, and / or computing units for processing data (e.g., computer programs). The processor can invoke the computer program stored in the memory to implement the multimodal cue fine-tuning method described in the above embodiments. Figure 4 Taking an electronic device consisting of one processor and one memory as an example, the processor and memory are used to indicate a type of device or equipment, and the quantity of each type of device or equipment can be determined according to business needs.
[0046] This application also provides an embodiment of a computer-readable storage medium, the computer-readable storage medium including a computer program or instructions that, when the computer program or instructions are run on a computer, cause the computer to perform the multimodal cueing fine-tuning method described in the above embodiment.
[0047] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of this application. Any modifications or equivalent substitutions that do not depart from the spirit and scope of this application should be covered within the protection scope of the claims of this application.
Claims
1. A multi-modal prompt fine-tuning method, characterized in that, include: Step 1: Obtain the dataset, divide the dataset into a training dataset and a test dataset, and divide the training dataset into a first training dataset and a second training dataset; Both the first training dataset and the second training dataset consist of image samples and their corresponding category labels; the category labels are used to indicate the category to which the image sample belongs; the number of image samples corresponding to each category in the second training dataset is less than the number of image samples corresponding to each category in the first training dataset; Step 2: Construct the network model, which includes a CLIP pre-trained text encoder, a CLIP pre-trained visual encoder, a cue vector selection network, and a projection network; Step 3: Input the first training dataset into the network model for training to obtain the first network model; input the second training dataset into the first network model for training to obtain the second network model; Step 4: Input the images from the test dataset into the second network model to obtain the image classification results, where the image classification results represent the category to which the image belongs; The step of inputting the first training dataset into the network model for training to obtain the first network model includes: Step 31: After segmenting all category labels in the first training dataset, input them into the CLIP pre-trained text encoder in the network model to obtain the text representation of each category label; Step 32: Input the text representation of each category label into the cue vector selection network in the network model to obtain a probability score; the probability score represents the matching degree between each cue vector in the cue vector pool and each category label; the cue vector is used to guide the network model to obtain the expected image classification result; Step 33: Based on the probability scores, obtain the category-specific cue vector corresponding to each category label; based on each category-specific cue vector and the CLIP pre-trained text encoder, obtain the text features of each category label in the first training dataset; Step 34: Input all image samples from the first training dataset into the CLIP pre-trained visual encoder in the network model to obtain the image representation of each image sample; Step 35: Obtain the visual features of each image sample based on the image representation of each image sample, the text representation of all the category labels, and the corresponding category-specific cue vectors; Step 36: Calculate the similarity between the visual features of each image sample and the text features of each current category label to obtain the category label corresponding to the maximum similarity; the category label is used to indicate the category to which each image sample belongs; Step 37: After executing steps 31 to 36, output the trained network model, i.e., the first network model; Step 33 includes: Step 331: Select the top k cue vectors with the highest probability scores, and use the top k cue vectors as the category-specific cue vectors corresponding to the category label; the category-specific cue vectors are the core semantic features most relevant to the category corresponding to the first training dataset; Step 332: Concatenate the category-specific prompt vector and its corresponding category name to obtain the text input vector; Step 333: Input the text input vector into the multi-layer Transformer encoder of the CLIP pre-trained text encoder to obtain the enhanced text representation, i.e., text features; Step 35 includes: Step 351: Calculate the similarity between the image representation of each image sample and the text representation of all the category labels to obtain multiple similarity scores; Step 352: Map the category-specific cue vectors of all the category labels to the visual space through the projection network in the network model to form intermediate visual cue vectors; Step 353: Use the multiple similarities as weights to weight the intermediate visual cue vectors in the intermediate visual cue vectors to obtain the final visual cue vector for each image sample; Step 354: Concatenate the final visual cue vector, category terminology, and image patch corresponding to each image sample, and input the concatenated vector into the CLIP pre-trained visual encoder to obtain the enhanced image representation, i.e., visual features; the category terminology is used to summarize the global features of each image sample.
2. The multimodal cue fine-tuning method according to claim 1, characterized in that, The CLIP pre-trained text encoder adopts a Transformer-based structure; the CLIP pre-trained visual encoder adopts a visual Transformer structure; the cue vector selection network is a multilayer perceptron, which consists of two linear layers and an activation layer located between them; the projection network is a feedforward neural network, which consists of linear layers.
3. The multimodal cue fine-tuning method according to claim 1, characterized in that, During the training of the network model and the first network model, the trainable parameters are optimized using cross-entropy loss; the trainable parameters include all cue vectors in the cue vector pool, cue vector selection network parameters, and projection network parameters; the backpropagation algorithm is used to optimize the trainable parameters; During the training of the cue vector selection network, KL divergence loss is used to ensure that the cue vector selection behavior between categories is consistent with their semantic similarity. This means that categories with greater similarity have a higher probability of selecting the same cue vector, thereby providing similar knowledge for similar categories and improving the ability of the network model and the first network model to adapt to new image classification tasks. The knowledge refers to the core semantic features most relevant to each category.
4. The multimodal cue fine-tuning method according to claim 3, characterized in that, The process of obtaining the KL divergence loss includes: Calculate the similarity between the text representations of each category label in the first training dataset and the second training dataset, and construct a similarity matrix from all the obtained similarities; Each category label from the first training dataset and the second training dataset is input into the prompt vector selection network to obtain the corresponding probability score; the similarity between the probability scores corresponding to each category label is calculated, and all the obtained similarities are used to form a distribution similarity matrix. The KL divergence loss is determined based on the similarity matrix and the distribution similarity matrix.
5. A multimodal cue fine-tuning system, implementing the multimodal cue fine-tuning method according to any one of claims 1-4, characterized in that, include: A dataset acquisition module is used to acquire a dataset, divide the dataset into a training dataset and a test dataset, and divide the training dataset into a first training dataset and a second training dataset. Both the first training dataset and the second training dataset consist of image samples and their corresponding category labels; the category labels are used to indicate the category to which the image sample belongs; the number of image samples corresponding to each category in the second training dataset is less than the number of image samples corresponding to each category in the first training dataset; The model building module is used to build a network model, which includes a CLIP pre-trained text encoder, a CLIP pre-trained visual encoder, a cue vector selection network, and a projection network. The model training module is used to input the first training dataset into the network model for training to obtain a first network model; and to input the second training dataset into the first network model for training to obtain a second network model. The image classification module is used to input the images in the test dataset into the second network model to obtain the image classification result, which is the category to which the image belongs.
6. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it implements the method of any one of claims 1-4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program or instructions that, when executed on a computer, cause the computer to perform the method of any one of claims 1-4.