Small sample learning method for task adaptive association learning

Abstract

The invention belongs to the technical field of image recognition, and relates to a small sample learning method for task-adaptive association learning, which comprises the following steps of: S1, assisting the original prompt by using a specific prompt word of a task; s2, enhancing the discrimination of the training features by using comparative learning; s3, using diversified image enhancement means to enhance the contrast of the image; according to the method, the generalization ability of the model is enhanced through rich description of a natural language, so that the CLIP can be more efficient even if the CLIP faces the unmet category or complex concept. Therefore, the method not only reduces the dependence on a large-scale annotation data set, but also improves the learning ability of the model under the condition of small samples and even zero samples, enhances the adaptability and generalization ability of the model to a new environment, greatly simplifies the deployment process of the model, and improves the deployment efficiency of the model. The method becomes a powerful tool for solving a series of challenges in image recognition and description tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and in particular relates to a small sample learning method for task-adaptive association learning. Background Art

[0002] With the rapid development of deep learning in the field of images, computer recognition of images has approached or even surpassed human performance. Although deep learning has achieved remarkable results in computer vision fields such as target recognition, scene understanding, and semantic segmentation, there are also some key bottleneck problems. For example, existing deep learning models rely heavily on centralized large-scale annotated training data. When annotated training data is scarce, the generalization of deep learning models will be significantly reduced, resulting in limited application scenarios. Unlike current deep learning models, humans generally only need a small amount of data to quickly master new target concepts. Inspired by this, researchers have proposed the concept of "small sample learning", which aims to explore how to use the knowledge and experience summarized from existing samples to solve new problems when the number of annotated samples of new target categories is very small.

[0003] Small sample learning method, that is, to improve the test accuracy with relatively few training samples. The specific method is to extract image features from the training data set through the CLIP model (Contrastive Language-Image Pre-Training, a large multimodal pre-training model proposed by openAI, which includes text and image encoders, defines positive samples as image-text pairs of the same category, and negative samples as image-text pairs other than these. The correspondence between the two modalities of image and text is learned through contrastive learning. The similarity between a given text and an image can be output.) Store them in a cache model, and define these image features as keys. At the same time, convert the labels corresponding to the training images into one-hot labels, called values. For a test image (which can be called a query), its label is predicted in two ways. The first is to calculate its similarity with the training image. If it is similar to a certain image, then the test sample is considered to belong to the category to which the image belongs. The specific calculation process is as follows: Figure 1 As shown in the horizontal flow, the corresponding prediction vector is obtained by dot-multiplying the test sample image features and the training sample image feature set, and then dot-multiplying the label set. The second is zero-shot reasoning through the CLIP model. First, the text features of all category names on the dataset are encoded to form W_c, and then dot-multiplied with the test image features to obtain the similarity, which is the prediction vector. Finally, the two prediction vectors are weighted and added, and the category index corresponding to the maximum logits is selected as the prediction category, as shown in Figure 1 shown.

[0004] However, the above small sample learning methods have the following defects:

[0005] Disadvantage 1: When this method uses CLIP to infer the test sample category, it uses the text embedding of all the category names in the dataset. However, encoding only the category name contains too little semantic information, because some category names may be very similar or have multiple meanings, which cannot accurately describe the sample and will cause misjudgment. For example, for a specific task, such as cat breed classification, the dataset contains all kinds of cats, so the category names may be very similar, which will cause the generated text features to be very similar, making it difficult to distinguish different categories. Or for the classification of some plant leaves, the category name is always "leaf", so the repeated appearance of the word "leaf" will increase category confusion and increase the difficulty of classification.

[0006] Disadvantage 2: This method simply stores the features of the training samples in the cache model without processing these features. The training samples for small sample tasks are very few, so the representation of each sample is crucial and related to the correct classification of the test sample. When encountering samples that are particularly similar visually, the extracted features are undoubtedly also similar. When the test sample searches for the most similar sample in such a feature set, the result obtained is bound to be inaccurate. Summary of the invention

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a small sample learning method for task-adaptive association learning, the method comprising the following steps:

[0008] S1. Use task-specific prompts to supplement existing prompts; specifically:

[0009] S1-1. Design templates for this task, including: Template 1 and Template 2. Template 1 includes: using the knowledge of experts in the field to generate more accurate prompts for the category, and Template 2 includes: using figurative and intuitive descriptors to directly describe the visual features presented by this category. By designing these two templates, it is possible to effectively avoid the generated prompts being general and containing abstract adjectives, which makes it impossible for CLIP to accurately understand and locate the appropriate position in the joint embedding space (the image and text feature embedding learned by CLIP through contrastive learning is an aligned joint embedding space, that is, the text and image of the same category are aligned in the embedding space), and then find the most similar image features.

[0010] S1-2, input the two templates designed in step S1-1 into the large language model in sequence, that is, first write the category name into template one and input it into LLM, you will get one or more prompt words, and then further input template two into LLM in a dialogue way, you will get the optimized prompt words; traverse all the categories in the data set, you can get a series of task-specific prompt words, save them, and use them in subsequent model reasoning;

[0011] S2. Use contrastive learning to enhance the discriminability of training features; specifically including:

[0012] S2-1. Design a linear layer network adapter, the size of which is consistent with the transpose of the size of the feature set, and initialize the network with the feature set extracted by the image feature extractor of CLIP; input the features of the input image into the adapter to obtain an affinity matrix affinity, which represents the similarity matrix between the input features and the training features; normalize affinity to obtain affinity_norm, take this variable as the object of optimization, and design a contrastive learning loss, which aims to shorten the distance between positive samples and push the distance between negative samples away, so as to learn useful feature representations;

[0013] S2-2, calculating the similarity matrix between the eigenvectors by multiplying the eigenvector matrix with its transpose and dividing by a temperature parameter; the temperature parameter can control the scale of the obtained similarity, a lower temperature will make the differences in the similarity matrix more significant, while a higher temperature will make these differences more gradual;

[0014] S2-3. Create a Boolean mask matrix mask with True on the diagonal to identify positive sample pairs (i.e., the similarity between each sample and itself). Use the mask to select the elements on the diagonal in the similarity matrix, i.e., positive samples, and the elements on the off-diagonal, i.e., negative samples. Combine the similarity values ​​of positive and negative samples into a logits tensor, where the positive sample is located at the first position of each row and the rest are negative samples.

[0015] S2-4. Create a labels tensor of all zeros, because the target of the positive sample is always the first position, corresponding to the first element of each row in the logits tensor. Use the cross entropy loss to calculate the final contrast loss value, where logits is used as the predicted value and the labels tensor of all zeros is used as the target value;

[0016] S3. Use a variety of image enhancement methods to enhance the contrast of the image; this is to increase the diversity of image data and improve the generalization ability of the model.

[0017] Preferably, in step S1-2, for different tasks or different data sets, the field needs to be specified in template 1 so that LLM can generate prompt words that best fit the task.

[0018] Preferably, in step S1, not only task-specific prompt words are used, but also the prompt template provided by the original and simplest CLIP is fully utilized to form prompt words; the prompt words of CLIP globally locate the category to the approximate position of the embedding space, while the task-specific prompt words are more refined local positioning, which can distinguish the differences between similar categories; step S1 adopts two weighted prompt words to jointly predict the category of the test sample, that is, the prediction vector obtained by the CLIP prompt words and the prediction vector obtained by the task-specific prompt words are weighted and added to form a prediction of visual-text association.

[0019] Preferably, in step S2, the contrast loss trains the model by minimizing the similarity difference between positive samples and negative samples, thereby learning to distinguish feature representations of different samples; the contrast loss is added to the loss of the entire model to optimize the feature distribution in the adapter, thereby obtaining a more accurate prediction.

[0020] Preferably, in step S1, the visual features and the corresponding labels are respectively represented as and

[0021] F = VisualEncoder(X K )

[0022] L=OneHot(Y N )

[0023] Among them, N represents N-type training samples, K represents the number of N-type training samples, and X K Indicates that there are N categories and K labeled images per category, Y N For label.

[0024] Preferably, in step S2, the similarity matrix is:

[0025]

[0026] Where τ is a temperature hyperparameter used to scale the similarity values;

[0027] The similarity between the positive sample and the negative sample is:

[0028] S + =S⊙M,S - =S⊙(1-M),

[0029] Where ⊙ represents element-wise multiplication, and 1 is a matrix of all 1s;

[0030] The contrast loss is:

[0031]

[0032] Where H is the cross entropy (CE) loss function.

[0033] The beneficial effects of the present invention are:

[0034] 1. The CLIP model used in the present invention demonstrates a deep understanding of cross-modal information by virtue of its unique ability to process image and text pairs; through prompt optimization, the recognition performance of CLIP is significantly enhanced; these prompts are like navigation lighthouses, guiding the model to lock on key visual features in the image and provide necessary contextual information, thereby effectively reducing uncertainty and ambiguity in the recognition process. More importantly, the present invention strengthens the generalization ability of the model through rich descriptions of natural language, so that CLIP can handle fine-grained categories or complex concepts with ease. Therefore, the method of the present invention not only reduces the dependence on large-scale annotated data sets, but also improves the learning ability of the model in small or even zero sample situations, and enhances the adaptability and generalization ability of the model in the face of new environments; by simply and cleverly using task-specific prompts, it is possible to adjust and optimize the performance of CLIP without changing the model architecture or retraining, greatly simplifying the deployment process of the model, making it a powerful tool for solving a series of challenges in image recognition and description tasks.

[0035] 2. The present invention has shown significant advantages in the field of small sample learning by adopting a contrastive learning loss function and optimizing the feature representation strategy. First, the contrastive learning loss of the present invention enhances the discriminability of features, effectively solves the problem of unclear sample boundaries in small sample sets, and greatly reduces the probability of misclassification between similar samples, thereby significantly improving the accuracy of classification. Secondly, the present invention realizes dynamic optimization of feature representation. This strategy not only improves the model's adaptability to new samples, but also effectively avoids overfitting, ensuring that the model still has strong generalization capabilities when the amount of data is limited. The present invention simplifies the storage requirements of the model, optimizes computing efficiency, and provides a more efficient and robust solution for small sample learning. It is particularly suitable for scenarios where data is scarce, and shows significant effects in improving model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a framework diagram of the Tip-Adapter of the prior art;

[0037] Figure 2 It is a model framework diagram of a small sample learning method for task-adaptive association learning of the present invention;

[0038] Figure 3 is a dimensionality reduction visualization diagram of text features of an embodiment of the present invention;

[0039] Figure 4 is a sample graph in the PlantDoc and PlantVillage datasets of the embodiment of the present invention;

[0040] Figure 5 4 is a comparison diagram of sample prediction results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the relevant technologies in the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] like Figures 1 to 5 As shown, Figure 1 This is the framework diagram of Tip-Adapter (Zhang, Renrui et al. “Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification.” European Conference on Computer Vision (2022).).

[0043] This embodiment and Figure 1 Compared with the method, this implementation has three improvements. The main idea is to increase visual relevance and multimodal relevance under the multimodal framework to improve the performance of the model:

[0044] 1. Use task-specific prompts (TSP) to supplement the original prompts.

[0045] The original CLIP-based prompt is: a photo of [CLS]. CLIP obtains the text embedding of the corresponding category by writing the category name into such a sentence and then feeding it into the text encoder. The present invention designs task-specific prompts for different tasks and categories. The specific operation is to first design a template for the task, such as "Use the knowledge of experts in this field to generate more accurate prompts for category [CLS]." (Template 1) and "Use figurative and intuitive descriptors to directly describe the visual features presented by this category." (Template 2) By designing these two templates, it is possible to effectively avoid the generated prompts being general and containing abstract adjectives, thereby preventing CLIP from accurately understanding and locating the appropriate position in the joint embedding space (the image and text feature embedding learned by CLIP through contrastive learning is an aligned joint embedding space, that is, the text and image of the same category are aligned in the embedding space), and then find the most similar image features. Secondly, the above two templates are input into the Large Language Models in sequence, that is, the category name is first written into Template 1 and input into LLM, and one or more prompt words will be obtained. Then Template 2 is further input into LLM in a dialogue manner, and the optimized prompt words will be obtained. By traversing all the categories in the dataset, a series of task-specific prompt words can be obtained, which are saved and used in subsequent model reasoning. It should be noted that for different tasks or different datasets, the field needs to be specified in Template 1 so that LLM can generate the prompt words that best fit the task.

[0046] In addition, the present invention not only utilizes task-specific prompt words, but also makes full use of the prompt template provided by the original and simplest CLIP to form prompt words. CLIP's prompt words globally locate the category to the approximate position of the embedding space, while task-specific prompt words are more refined local positioning, which can distinguish the differences between similar categories. Therefore, the present invention adopts two weighted prompt word modes to jointly predict the category of the test sample, that is, the prediction vector obtained by the prompt word of CLIP and the prediction vector obtained by the task-specific prompt word are weighted and added to form the prediction of visual-text association.

[0047] 2. Using contrastive learning to enhance the discriminability of training features

[0048] Since small sample learning is extremely dependent on very few training samples, the representation ability of the training samples is directly related to the prediction accuracy of the test data. Compared with the aforementioned literature that directly saves the image feature set of the training samples to query and compare with the features of the test samples, the present invention proposes to enhance the features of the feature set to increase the discriminability between the features. First, a linear layer network adapter is designed, the size of which is consistent with the transpose of the size of the feature set, and the network is initialized using the feature set extracted by the image feature extractor of CLIP. (This step is consistent with Tip-Adapter) Inputting the features of the input image into the adapter can obtain an affinity matrix affinity, which represents the similarity matrix between the input features and the training features. Normalizing affinity to obtain affinity_norm, taking this variable as the object of optimization, a contrastive learning loss is designed to narrow the distance between positive samples and push the distance between negative samples away, thereby learning useful feature representations. Secondly, the similarity matrix between the feature vectors (affinity) is calculated by multiplying the feature vector matrix with its transpose and dividing it by the temperature parameter temperature. The temperature parameter controls the scale of the resulting similarity, with lower temperatures making the differences in the similarity matrix more pronounced, while higher temperatures make these differences more gradual. Then create a Boolean mask matrix mask with True on the diagonal to identify positive sample pairs (i.e., the similarity of each sample to itself), and select the elements on the diagonal (positive samples) and the elements on the off-diagonal (negative samples) in the similarity matrix through the mask. Combine the similarity values ​​of the positive and negative samples into a logits tensor, where the positive samples are located in the first position of each row and the rest are negative samples. Create an all-zero labels tensor, because the target of the positive sample is always the first position, corresponding to the first element of each row in the logits tensor. Use the cross entropy loss to calculate the final contrastive loss value, with logits as the predicted value and the all-zero labels tensor as the target value. In general, this contrastive loss trains the model by minimizing the difference in similarity between positive and negative samples, thereby learning to distinguish the feature representations of different samples. This contrastive loss is added to the loss of the entire model to optimize the feature distribution in the adapter, so that more accurate predictions can be obtained. The network itself also has a classification loss, which is the prediction vector (Cache logits ) and the prediction vector of multimodal association (ω*CLIP logits +(1-ω)*TSP logits ) and the cross entropy loss between the predicted vector and the true label.

[0049] 3. Task-Specific Image Enhancement

[0050] For specific tasks, if visually similar samples are included, ordinary data enhancement methods cannot meet the requirements. For example, the default image enhancement of CLIP only includes two operations: scaling and cropping. Therefore, we propose to use a variety of image enhancement methods. And we propose to enhance only the training data. Combined with the second improvement above, it can effectively improve the discriminative representation ability of the training sample features. Enhance the contrast of the image. This is to increase the diversity of image data and improve the generalization ability of the model.

[0051] The key points of this implementation are: 1. Task-specific prompts adjust and optimize the performance of CLIP on small sample recognition tasks. 2. Comparative learning supervised training feature optimization reduces the probability of misclassification between similar samples. 3. Data augmentation technology can enrich training samples.

[0052] Example

[0053] This embodiment is to solve the problem of fine-grained small sample classification in plant disease identification, and proposes an adaptive relationship learning method to learn the visual and language recognition relationship between disease categories. The framework of the method in this embodiment is as follows Figure 2 As shown. In general, this embodiment uses the visual and text encoders of frozen CLIP to extract features of images and labels, and uses a learnable linear layer adapter to store and fine-tune the features of a small sample training set. The prediction of this network consists of two parts: visual similarity mapping and cross-modal semantic alignment. The visual similarity mapping depends on the uniqueness of the training set features, so this embodiment introduces a contrast method to learn more discriminative features. In terms of cross-modal semantic alignment, this embodiment uses TSP, which supplements the plant disease prediction task with specific prior information to improve the accuracy of fine-grained classification. At the same time, this embodiment performs contrast enhancement on the training images to improve the robustness and generalization of visual similarity. Finally, this embodiment proposes a fusion prediction of TSP and hand-crafted hint computation predictions.

[0054] Visual Similarity Mapping

[0055] The existing methods simply store image features in a memory bank for calculating the similarity between test samples. These methods focus on coarse-grained classification and ignore the inter-class feature similarity for fine-grained classification. This embodiment constructs a key-value cache model by extracting CLIP features from small sample data to optimize training features and improve the visual similarity mapping between training samples and test samples. Given a small sample learning dataset containing K N-class training samples, including N categories and K labeled images per category, denoted as X K , labelled as Y NWe built a key-value cache model to adapt the training features to the CLIP model, which contains small sample knowledge about N classes. For the training samples, this example uses CLIP to extract C-dimensional L2-normalized visual features and convert the true value labels into one-hot vectors. The visual features and corresponding labels are represented as and

[0056] F = VisualEncoder(X K )

[0057] L=OneHot(Y N )

[0058] This embodiment Figure 2 The labels are hidden in the cache model in . Visual features are considered as keys of the key-value cache model, and labels are considered as values. The knowledge extracted from the small sample learning dataset is stored in the cache model for later fine-tuning.

[0059] In order to improve the discriminability of the cached model, this embodiment introduces a self-contrast method. First, we calculate the affinity between the training features and the test features. The C-dimensional visual features of the test image extracted by CLIP can be expressed as as a query to retrieve the most similar samples from the cached model. The affinity between the query (test feature) and the key (training feature) can be calculated by matrix-vector multiplication:

[0060] A=exp(-β(1-fF T )),

[0061] in and β represent modulation hyperparameters. In the case of a few samples, the amount of data is very limited, and directly optimizing the training features may be difficult to capture the complex patterns and changes between the data. On the contrary, by optimizing the similarity (affinity) between samples, the model can better utilize the contrast information between these limited samples and effectively utilize the supervisory signal provided by each sample instance, thereby achieving more effective learning on a small number of samples.

[0062] Then, this embodiment constructs an affinity comparison strategy. In order to bring positive sample pairs closer together and negative sample pairs farther apart, this embodiment constructs a self-similarity matrix:

[0063]

[0064] where τ is a temperature hyperparameter used to scale the similarity values.

[0065] Then, this embodiment constructs an indicator function for positive samples and negative samples. A positive sample pair consists of features of the same sample, usually defined by a mask M∈{0,1} N×N Among them, M ii =1 represents the diagonal elements (i.e., positive sample pairs), and the rest are zero. Negative sample pairs are all non-diagonal elements.

[0066] This embodiment extracts the similarity of positive samples and negative samples respectively:

[0067] S + =S☉M,S - =S☉(1-M),

[0068] Where ⊙ represents element-wise multiplication, and 1 is a matrix of all 1s. Merge the similarity of positive and negative samples:

[0069]

[0070] Where B is the batch size and the label vector is set tar get i =[0,0,0,…,0].

[0071] Finally, in order to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs, the self-contrastive loss can be expressed as:

[0072]

[0073] Where H is the cross entropy (CE) loss function.

[0074] The logarithmic position in CE is the concatenation of the similarities of positive and negative samples. For each sample, the first element of its logarithmic vector is the similarity with itself (i.e., the positive sample), and the remaining elements are the similarities with other samples (i.e., negative samples). The target label target is a vector of all zeros. This is because, for each sample, this embodiment hopes that the first element of the logarithmic vector (i.e., the similarity with itself, i.e., the positive sample) is the maximum value, so that the similarity of each sample with itself is the maximum value, and the similarity with other samples is the minimum value.

[0075] In addition, in order to enhance the model's perception of the visual features of leaves, this embodiment also proposes a customized data enhancement technology, namely, Contrastive Enhancement Augmentation (CEA), to address two key challenges inherent in traditional visual encoders: one is its general design, which may not be able to fully utilize the unique features of plant leaves; the other is the robustness of feature representation in the cache model, which is particularly important when samples are limited.

[0076] The theoretical rationale behind CEA is that standard visual encoding architectures are not designed for plant diseases and suffer from performance gaps due to inability to optimally detect complex details and changes in diseased leaves. By enhancing the contrast of leaf images, CEA can emphasize the visual boundaries between healthy and diseased tissue, enriching the discriminative information available to the model. As a result, this enhancement promotes a more nuanced understanding of disease symptoms and improves the model's ability to detect subtle disease indicators.

[0077] In addition, CEA is also crucial for enhancing feature robustness, especially when the number of samples is small. It can improve the contrast of training images, thereby enhancing feature diversity and maximizing the information extracted from each sample, which is crucial for datasets with sparse data. This will improve generalization ability, reduce the risk of overfitting, and improve accuracy in low-data environments.

[0078] Enhance ImageX aug It is achieved by applying a contrast enhancement function f to the pixel intensity values ​​of the original image X (for example) CEA Defined as

[0079] X aug =f CEA (X) = α(X-μ) + μ,

[0080] where μ is X and α is a parameter that controls the degree of contrast enhancement.

[0081] Cross-modal semantic alignment

[0082] This example uses an innovative prompt design method called Task Specific Prompts (TSP), which aims to optimize the performance of visual language models in small-sample learning tasks by integrating fine-grained category information into the prompt construction process, enriching text embeddings with flexible semantic scope, and improving cross-modal semantic alignment. Unlike traditional hand-crafted prompts, the TSP strategy is specifically designed to focus on and learn important features of specific categories.

[0083] This embodiment specifically designs some templates for plant disease identification. Researchers can use these templates in sequence, that is, use the first template to generate prompts, and then use the next template to optimize the generated prompts. The templates designed in this embodiment are as follows:

[0084] Leverage the knowledge of domain experts to build more accurate prompts for [CLS].

[0085] Use graphic and intuitive descriptions to directly describe the visual characteristics of this plant disease.

[0086] [CLS] represents the class of disease to be described.

[0087] To address the challenges of the specific task of plant disease identification, each prompt in ACIP is carefully designed around a specific plant disease to ensure the relevance and accuracy of the description. For example, for "apple scab leaves", the prompt used in this example is "a photo of apple scab leaves showing dark, rough, scab-like lesions characteristic of apple scab" to accurately convey the unique characteristics of apple scab. As for "tomato early blight leaves", this example uses "a picture depicting early-appearing ring-shaped black spots on tomato leaves" to accurately focus on the symptoms of early blight. Through this type of description, TSP not only summarizes the general appearance of the category, but also delves into the specific visual characteristics of the disease, such as color, shape, and distribution patterns, which are key to distinguishing different disease types.

[0088] exist Figure 3 In this example, we visualize the text embeddings of all categories in the PlantDoc dataset under two types of prompts: CLIP handcrafted prompts and TSP prompts. After TSP optimization, the text features become more discriminative, which shows that extending the category semantics can improve the classification potential. This example highlights the feature distribution of categories with different confusion levels before and after prompt optimization, and Figure 3 The original confused samples are provided at the bottom. Pentagons represent severely confused categories. With the handcrafted prompts, the features of "tomato early blight leaves" and "tomato late blight leaves" almost overlap, but the specific disease description significantly increases the distance. Similarly, for the moderately confused "tomato moldy leaves" and "tomato leaves", the improved prompts increase the feature distance by about 8 times. Even for samples with a lower degree of confusion (mild), TSP can further increase the distance between categories. These results show that the prompt-optimized TSP of this embodiment can effectively enhance the discriminability of text features and improve the model's ability to distinguish visually similar plant diseases.

[0089] TSP directly maps complex disease features in images that are difficult to intuitively identify through pixels through language, which prompts the model to go beyond the pixel level when understanding images and directly reach the essential attributes of the disease. In this way, TSP not only enriches the semantic information received by the model, but also promotes the close integration of visual features and highly relevant language descriptions.

[0090] Fusion prediction of multimodal models

[0091] For the test image, this embodiment uses CLIP to extract C-dimensional visual and text features, which are respectively denoted as and In addition, this embodiment sets a category prompt, which can obtain TSP-guided text embedding Then, the calculation formula for prediction by visual-text similarity is

[0092]

[0093] Besides, the few-shot knowledge retrieved from the cached model also helps predict visual-visual similarity.

[0094]

[0095] in

[0096] Finally, this embodiment integrates the two predictions to obtain a fused prediction:

[0097] logits=logits text +γlogits visual

[0098] Where γ represents the residual ratio. The final predicted category is the category corresponding to the position with the maximum value of logits.

[0099] This example conducts experiments on two widely used plant disease recognition datasets: Figure 4 Samples of both datasets are shown. The PlantDoc dataset was collected under natural field conditions and contains 2,569 images of 13 crops in 27 categories (17 disease categories and 10 healthy categories). In contrast, the widely used PlantVillage dataset, launched by Pennsylvania State University for plant disease diagnosis, contains 54,305 images of healthy and diseased leaves of 14 crop varieties, divided into 38 categories (26 diseases and 12 health). However, an important limitation of the PlantVillage database is that the images were not taken in natural environments, but were collected in laboratory settings with plain backgrounds. Figure 5 As shown in Figure 2, PlantDoc consists of diseased plant images with different environments, multiple leaves, fruits, and different light and illumination conditions. PlantVillage images have clear backgrounds and uniform capture and illumination conditions. Figure 5 Samples from the PlantDoc and PlantVillage datasets. The PlantDoc dataset is taken from a natural environment and is more challenging due to the presence of various backgrounds and leaves. The PlantVillage dataset was collected in a laboratory environment with a simple background and only one leaf in one image.

[0100] Experimental setup

[0101] In terms of small sample learning, this embodiment compares the performance of training sets of 1, 2, 4, 8, and 16 samples per class, and tests on the full test set. This embodiment selects ResNet-50 as the visual encoder of the CLIP backbone and Transformer as the text encoder. This embodiment uses a pre-trained CLIP model and freezes the parameters during training. Image enhancement is consistent with the preprocessing protocol of CLIP, which includes random cropping, resizing, and random horizontal flipping. The contrast enhancement parameter α is set to 0.5. The parameter search function automatically learns β. The weighting parameter ω is set to 0.5. The temperature parameter τ and weight parameter γ of the learning loss are compared, and τ is finally selected as 0.5 and γ is 0.5. The learnable cache model is a linear layer with input and output sizes as training features (keys) and initialized by the transpose of the keys. During the training and testing phases, the layer is fine-tuned using batch sizes of 64 and 256, a learning rate of 0.001, and an AdamW optimizer with a cosine scheduler, respectively. For most results, this embodiment sets the training epoch to 40.

[0102] Depending on whether the cache model is learnable, this example compares two methods: training-free (CALP) and training-required (CALP-F, CALP-Finetune). For the former, visual-visual similarity is directly calculated from affinity, while for the latter, the cache model is fine-tuned. This example also compares the results of using only TSP for prediction (called "new hint") and using fused prediction.

[0103] This example compares the performance of zero-shot CLIP, Tip-Adapter, and Tip-Adapter-F (Tip-Adapter with fine-tuning). Zero-shot CLIP uses pre-trained knowledge for classification and does not require additional training. Tip-Adapter stores training features and uses them for prediction. Tip-Adapter-F converts training features into learnable features. The classification results (classification accuracy, expressed as a percentage) of the PlantDoc dataset are shown in Table 1:

[0104] method 1 2 4 8 16 24 36 48 64 128 Zero-shot CLIP 11.46 11.46 11.46 11.46 11.46 11.46 11.46 11.46 11.46 11.46 Zero-shot CLIP + new tips 19.90 19.90 19.90 19.90 19.90 19.90 19.90 19.90 19.90 19.90 Zero-shot CLIP+ fusion prediction 17.23 17.23 17.23 17.23 17.23 17.23 17.23 17.23 17.23 17.23 Tip-Adapter 13.89 20.37 31.73 42.04 55.19 59.62 59.14 61.73 59.37 59.42 CALP+ New Tips 25.01 32.08 39.77 47.84 55.21 59.09 57.02 59.75 58.22 57.37 CALP+ fusion prediction 23.92 37.78 54.23 56.27 56.90 60.25 59.05 61.18 59.96 58.91 Tip-Adapter-F 34.41 44.72 60.08 70.30 77.84 80.48 83.74 85.64 86.90 90.84 CALP-F+ New Tips 28.48 38.96 45.77 66.38 75.23 82.04 84.40 85.71 87.49 90.82 CALP-F+ fusion prediction 37.80 54.06 65.05 75.72 79.17 82.38 84.66 86.28 87.66 91.66

[0105] Table 1 The results of the PlantVillage dataset are shown in Table 2:

[0106] method 1 2 4 8 16 24 36 48 64 128 Zero-shot CLIP 24.27 24.27 24.27 24.27 24.27 24.27 24.27 24.27 24.27 24.27 Zero-shot CLIP + new tips 28.93 28.93 28.93 28.93 28.93 28.93 28.93 28.93 28.93 28.93 Zero-shot CLIP+ fusion prediction 28.35 28.35 28.35 28.35 28.35 28.35 28.35 28.35 28.35 28.35 Tip-Adapter 26.21 29.71 30.29 40.58 41.75 43.88 44.66 44.75 45.05 48.74 CALP+ New Tips 29.71 34.76 33.40 42.91 42.33 47.38 45.24 44.36 46.80 46.41 CALP+ fusion prediction 28.35 34.95 32.62 43.11 44.66 47.18 45.44 44.75 46.80 47.38 Tip-Adapter-F 37.48 36.89 38.45 50.29 55.34 61.75 61.17 58.83 63.50 64.08 CALP-F+ New Tips 38.25 40.97 40.19 46.60 57.28 60.58 62.14 59.42 63.69 61.94 CALP-F+ fusion prediction 42.14 42.52 42.33 53.20 60.19 63.11 63.11 61.55 64.47 65.63

[0107] Table 2

[0108] Some classification results: Figure 5Comparison of sample prediction results for three methods: Tip-Adapter, CALP Tips, and CALP Fusion. Dark black bars are correct predictions, light black red bars are incorrect predictions, and gray bars are other predictions. The labels of the top 3 predictions are shown.

[0109] In summary, the present invention strengthens the generalization ability of the model through rich descriptions in natural language, so that CLIP can handle fine-grained categories or complex concepts with ease. The method of the present invention not only reduces the dependence on large-scale annotated datasets, but also improves the learning ability of the model in small or even zero sample situations, and enhances the adaptability and generalization ability of the model in the face of new environments. The present invention greatly simplifies the deployment process of the model, making it a powerful tool for solving a series of challenges in image recognition and description tasks.

[0110] It should be emphasized that the above are only preferred embodiments of the present invention and do not limit the present invention in any form. Any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A small sample learning method for task-adaptive association learning, characterized in that: The method comprises the following steps: S1. Use task-specific prompts to supplement existing prompts; specifically: S1-1. Design a template for the task, the template comprising: Template 1 and Template 2. Template 1 comprises: utilizing the knowledge of experts in the field to generate more accurate prompts for the category; Template 2 comprises: using figurative and intuitive descriptive words to directly describe the visual features presented by the category; S1-2, input the two templates designed in step S1-1 into the large language model in sequence, that is, first write the category name into template one and input it into LLM, you will get one or more prompt words, and then further input template two into LLM in a dialogue way, you will get the optimized prompt words; traverse all the categories in the data set, you can get a series of task-specific prompt words, save them, and use them in subsequent model reasoning; S2. Use contrastive learning to enhance the discriminability of training features; specifically including: S2-1. Design a linear layer network adapter, the size of which is consistent with the transpose of the size of the feature set, and initialize the network using the feature set extracted by the CLIP image feature extractor; input the features of the input image into the adapter to obtain an affinity matrix affinity, which represents the similarity matrix between the input features and the training features; normalize affinity to obtain affinity_norm, and use this variable as the object of optimization; S2-2, calculating the similarity matrix between the eigenvectors by multiplying the eigenvector matrix by its transpose and dividing by the temperature parameter; S2-3. Create a Boolean mask matrix mask with True on the diagonal. Use the mask to select the elements on the diagonal of the similarity matrix, i.e., positive samples, and the elements on the off-diagonal, i.e., negative samples. Combine the similarity values ​​of the positive and negative samples into a logits tensor, where the positive samples are located at the first position of each row and the rest are negative samples. S2-4. Create a zero labels tensor and use the cross entropy loss to calculate the final contrast loss value, where logits are used as the predicted value and the zero labels tensor is used as the target value. S3. Use a variety of image enhancement methods to enhance the contrast of the image.

2. The small sample learning method for task-adaptive association learning according to claim 1, characterized in that: In step S1-2, for different tasks or different data sets, the field needs to be specified in template 1 so that LLM can generate prompt words that best fit the task.

3. The small sample learning method for task-adaptive association learning according to claim 1, characterized in that: In the step S1, not only the task-specific prompt words are used, but also the prompt template provided by the original and simplest CLIP is fully utilized to form the prompt words; the prompt words of CLIP globally locate the category to the approximate position of the embedding space, while the task-specific prompt words are more precise local positioning; the step S1 adopts a weighted method of the two prompt words to jointly predict the category of the test sample, that is, the prediction vector obtained by the CLIP prompt words and the prediction vector obtained by the task-specific prompt words are weighted and added to form a prediction of visual-text association.

4. The small sample learning method for task-adaptive association learning according to claim 1, characterized in that: In step S2, the contrast loss trains the model by minimizing the similarity difference between positive samples and negative samples, thereby learning to distinguish feature representations of different samples; the contrast loss is added to the loss of the entire model to optimize the feature distribution in the adapter, thereby obtaining a more accurate prediction.

5. The small sample learning method for task-adaptive association learning according to claim 1, characterized in that: In step S1, the visual features and the corresponding labels are respectively represented as and F=VisualEncoder(X K ) L=OneHot(Y N ) Among them, N represents N-type training samples, K represents the number of N-type training samples, and X K Indicates that there are N categories and K labeled images per category, Y N For label.

6. The small sample learning method for task-adaptive association learning according to claim 1, characterized in that: In step S2, the similarity matrix is: Where τ is a temperature hyperparameter used to scale the similarity values; The similarity between the positive sample and the negative sample is: S + =S⊙M,S - =S⊙(1-M), Where ⊙ represents element-wise multiplication, and 1 is a matrix of all 1s; The contrast loss is: Where H is the cross entropy CE loss function.

Images