Image classification method based on attention mechanism and context optimization prompt

Abstract

The invention belongs to the field of computer vision, discloses a context optimization prompt algorithm and system based on an attention mechanism, computer equipment and application thereof, and aims to improve the performance and generalization ability of a vision-language model through prompt fine tuning. The method comprises the following steps: firstly, obtaining a to-be-processed text prompt and corresponding image data, generating diversified prompt words by utilizing a context optimization mechanism, and inputting the diversified prompt words and an image into a CLIP model for multi-level feature extraction; and then, feature expression is optimized through adaptive weight distribution, information of diversified cue words is fused in combination with an attention mechanism, and a final classification loss function is calculated. And adjusting model parameters according to the loss function, optimizing a prompt strategy, and completing an image classification task and obtaining a classification result by taking minimization of classification loss as a target. According to the method, the accuracy of image classification is remarkably improved, and the generalization ability of the model to unknown categories is enhanced especially under the condition that the data volume is limited. By introducing an attention mechanism, the model can capture semantic information in the image more accurately, the overfitting of a basic category is reduced, and the method is suitable for being applied in a resource-limited environment.

Description

Technical Field

[0001] This invention relates to the fields of image processing and deep learning technology, and in particular to a few-shot image classification method based on fine-tuning of a vision-language pre-trained model and optimization of the attention mechanism. Background Technology

[0002] Currently, with the rapid development of artificial intelligence and computer vision technologies, visual-language pre-trained models play a crucial role in image classification tasks. These models, pre-trained on large-scale image-text datasets, learn rich visual and linguistic feature representations and demonstrate superior performance in multiple visual classification tasks. Among them, the CLIP (Contrastive Language-Image Pre-training) model is a typical example. Through a contrastive learning strategy, it successfully trains an image encoder and a text encoder, enabling cue-based zero-shot classification in various visual tasks. The CLIP model uses a large-scale dataset containing 400 million image-text pairs, exhibiting excellent performance in aligning image and text features, significantly improving the model's generalization ability.

[0003] While existing vision-language pre-trained models perform well on multimodal tasks, they still have limitations in adapting to specific downstream tasks. For example, hand-designed cue templates in the CLIP model (such as "a photo of a [CLASS]") can achieve zero-shot prediction, but their flexibility and adaptability in handling different tasks are limited due to a lack of deep understanding of task-specific knowledge. To improve the model's performance on specific tasks, researchers have proposed cue fine-tuning methods based on few-shot learning, such as the CoOp (Context Optimization) method. CoOp improves the model's inference ability by replacing hand-designed cue with learnable soft cue. However, the CoOp method has an inherent drawback: its generated cue is fixed for each task and fails to fully consider the feature differences between different images. To address this issue, the CoCoOp (Conditional Context Optimization) method enhances the model's generalization ability by generating specific image conditional contexts for each image and combining them with text conditional contexts. However, although this method improves performance on unknown categories, it suffers a decrease in base class recognition accuracy, which is a significant drawback. Furthermore, attention mechanisms have become a crucial technique in visual-language models. By dynamically focusing relevant information, attention mechanisms significantly enhance the model's ability to understand and represent multimodal data. This mechanism typically transforms textual features into representations of queries, keys, and values, and obtains attention scores by calculating the dot product between queries and keys. These scores are then normalized and combined with the values ​​to generate context-sensitive feature representations. The introduction of attention mechanisms not only enhances the feature integration capabilities of visual-language models but also provides strong support for their application in tasks such as image description generation, visual question answering, and zero-shot learning.

[0004] In summary, although existing vision-language pre-trained models demonstrate strong capabilities in multimodal tasks, they still have shortcomings in terms of adaptability to specific downstream tasks and stability in basic class recognition. Specifically, existing technologies cannot effectively balance the target task recognition accuracy and the generalization accuracy of large multimodal models in downstream classification tasks; that is, they cannot achieve both high accuracy in target task recognition and excellent generalization ability for unknown tasks. Summary of the Invention

[0005] To address the problems of existing technologies, this invention proposes a context-based fine-tuning method using an attention mechanism, specifically designed for few-shot image classification tasks. Context-optimized fine-tuning improves the model's classification accuracy with limited samples, while feature fusion using an attention mechanism effectively enhances the model's generalization ability.

[0006] This method mainly includes the following key technical steps:

[0007] Step 1: Utilize the pre-trained visual-language large model CLIP as a foundation. This model learns rich visual and linguistic feature representations by pre-training on a large-scale image-text dataset.

[0008] Step two involves introducing a context-optimized prompting strategy. This strategy combines trainable text prompts with category labels to acquire textual knowledge about images of different categories. Specifically, it introduces learnable prompt words to fine-tune the CLIP model. By freezing the parameters of the visual text encoder, the learnable prompt words are used to adapt CLIP to downstream tasks.

[0009] Step three involves fusing text features using an attention mechanism. An attention network is constructed by designing linear transformations of the query (Q), key (K), and value (V), and a Softmax function is used to generate standardized attention weights, enabling effective filtering and integration of key information. Then, a feature fusion strategy is introduced after the text encoder to combine contextual features with the original text features, and further transforms them through linear layers to produce the final feature representation.

[0010] Step 4: Design a contrastive loss function, combine it with standard cross-entropy loss, and optimize the model's ability to distinguish between known and unknown tasks by adjusting hyperparameters to balance the recognition performance of different categories.

[0011] Step 5: Use the obtained attention-based context optimization model to classify images in the image dataset or in the real world;

[0012] Furthermore, the introduction of the context-optimized suggestion strategy in step two includes the following steps:

[0013] The initial feature extraction layer in the CLIP vision-language model comprises a visual encoder and a text encoder. The visual encoder (defined as α) maps a given image (I) to the visual embedding space, generating initial support feature maps and initial query feature maps, i.e., the visual feature embedding x = α(I). Simultaneously, the text encoder (defined as β) embeds text into the text embedding space. Specifically, the text embedding... The process involves the following steps: First, the text prompt "a photo of a [CLASSNAME]." of category [CLASSNAME] is encoded using the Transformer encoder γ(.). The formula is as follows:

[0014]

[0015] Then Input text encoder β, generate text embedding

[0016] A CLIP feature fusion layer is constructed to generate prototype and adaptive feature maps. This is achieved through visual feature embeddings and text embeddings. Similarity matching, calculate the predicted probability p clip (y|x), the formula is as follows:

[0017]

[0018] Where d represents the cosine similarity. It is a learnable temperature parameter.

[0019] A feature matching layer is constructed for a few-sample image classification method, and the CoOp model is introduced to optimize the CLIP prompts, generating a matching score tensor and class prototypes. CoOp first defines a set of continuous text vectors V = {v1, v2, v3, v4, v5}. 2, …,v M} as a learnable c, and embed category words into c i The final text embedding is generated by connecting with learnable hints v. The formula is as follows:

[0020]

[0021] By incorporating a small number of samples from downstream tasks, CoOp minimizes the image feature x and the category text embedding. The negative log-likelihood between the two vectors is used to optimize the learnable vector V, as shown in the following formula:

[0022]

[0023] Throughout the process, both the visual encoder and the text encoder remain frozen, and CoOp infers appropriate task-related cues. This is to enhance the model's generalization and discrimination capabilities.

[0024] Furthermore, step three employs an attention mechanism to fuse text features, including the following steps:

[0025] We construct a feature extraction layer in an attention-based text feature fusion method and use this layer to generate initial feature representations. First, we embed z into the input text features. y Perform a linear transformation to generate representations of the query (Q), key (K), and value (V). The formula is as follows:

[0026] Q = W Q ·z y K = W K ·z y V = W V ·z y (5)

[0027] Among them, W Q W K W V These represent the corresponding linear transformation weight matrices.

[0028] Then, the dot product between query Q and key K is calculated, scaled, and an attention score is generated. Normalized attention weights are then generated using the Softmax function, as shown in the following formula:

[0029]

[0030] By performing a weighted summation of normalized attention weights and the value V, a context-dependent feature representation z is generated. f The formula is as follows:

[0031] z f =Attention·V (7)

[0032] Finally, the context features z f Compared with the original text features z y Combined, further feature transformation is performed through a linear transformation layer to generate the final fused feature representation Z. fused The formula is as follows:

[0033] Z fused =W f ·[z f ,z f (8)

[0034] Furthermore, step four, concerning the design of the loss function, includes the following steps:

[0035] We design a model optimization method based on a contrastive loss function. First, the text embeddings of hand-crafted prompts and the learnable text embeddings are defined as follows: and The text embeddings of manually prompted words and learnable text embeddings are used as the query (Q), key (K), and value (V) inputs to the attention mechanism module, respectively, to obtain the fused text embedding Z. fused The formula is as follows:

[0036] Z fused =Attention(z) coop ,z clip ,z clip (9)

[0037] Next, the standard contrast loss L used to fit the base classes is calculated. coop The formula is as follows:

[0038]

[0039] Finally, the contrastive loss L, used to prevent the model from overfitting to the base class, is calculated. attention The formula is as follows:

[0040]

[0041] By combining standard cross-entropy loss and tuning hyperparameters, the model's discriminative ability in both known and unknown tasks can be optimized. The standard cross-entropy loss L... coop Contrast loss L to prevent overfitting attention Combining these, we obtain the final target loss function L, as shown in the following formula:

[0042] L=(1-θ)L coop + θ L attention (12)

[0043] Here, θ is a hyperparameter used to balance the effects of the two losses in the final target task. By adjusting the hyperparameter θ, the model's recognition performance in different categories is balanced, thereby optimizing the model's discriminative ability in both known and unknown tasks.

[0044] The technical solution of this invention has the following advantages and positive effects: Through fine-tuning the CLIP template, this invention significantly improves the accuracy of image classification, especially when the amount of data is limited, the model can still maintain high classification accuracy. The introduction of an attention mechanism further enhances the model's generalization ability to unknown categories, enabling the model to better capture semantic information in images and reduce overfitting to basic categories. The few-shot training method reduces the need and cost of manual annotation, and thanks to the improved performance of the model in few-shot learning, it reduces the dependence on a large amount of labeled data. Furthermore, this invention can still achieve high classification accuracy even with very few samples, such as single samples, significantly improving the model's computational efficiency and making it suitable for resource-constrained environments.

[0045] The specific embodiments of the present invention will now be described in further detail with reference to the accompanying drawings. Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become more apparent from the description, or may be realized by practice of the invention. Attached Figure Description

[0046] Figure 1 This is a flowchart of the context optimization prompting algorithm based on the attention mechanism in Embodiment 1 of the present invention.

[0047] Figure 2 This is a network structure diagram of the context-optimized prompting algorithm based on an improved attention mechanism for image classification in Embodiment 1 of the present invention.

[0048] Figure 3 This is a schematic diagram comparing the classification accuracy of the present invention with that of three other existing models in the case of a small number of samples, as described in the second embodiment of the present invention.

[0049] Figure 4 This is a schematic diagram illustrating the effectiveness of the attention-based feature fusion module introduced in the third embodiment of the present invention in image classification tasks. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0051] like Figure 1 As shown, this embodiment of the invention discloses a context-optimized prompting method based on an attention mechanism, comprising the following steps:

[0052] S101: Obtain the text prompts to be processed and the corresponding image data, i.e., each image and its corresponding label;

[0053] S102: Two sets of text prompt features are obtained using a frozen CLIP text encoder, using both a hand-crafted prompt template and a learnable prompt template.

[0054] S103: Input the image into the image encoder in the CLIP model to extract image features;

[0055] S104: Use an attention mechanism to fuse the features of the two sets of text features and calculate the final classification loss function;

[0056] S105: Based on the classification loss function, adjust the model parameters and optimize the prompting strategy to achieve the minimum classification loss function, complete the image classification task, and obtain the classification result.

[0057] The attention-based context optimization prompting algorithm provided in this invention enables fine-tuning and optimization of the CLIP model. The network structure diagram of its training process is shown below. Figure 2 As shown, the specific steps include:

[0058] Step 1: Utilize the pre-trained visual-language large model CLIP as a foundation. This model learns rich visual and linguistic feature representations by pre-training on a large-scale image-text dataset.

[0059] Step two involves constructing the feature extraction layer in the initial model of the visual-language model CLIP, including a visual encoder and a text encoder. The visual encoder (defined as α) maps a given image (I) to the visual embedding space, generating initial support feature maps and initial query feature maps, i.e., the visual feature embedding x = α(I). Simultaneously, the text encoder (defined as β) embeds text into the text embedding space. Specifically, the text embedding... The process involves the following steps: First, the text prompt "a photo of a [CLASSNAME]." of category [CLASSNAME] is encoded using the Transformer encoder γ(.). The formula is as follows:

[0060]

[0061] Then Input text encoder β, generate text embedding

[0062] A CLIP feature fusion layer is constructed to generate prototype and adaptive feature maps. This is achieved through visual feature embeddings and text embeddings. Similarity matching, calculate the predicted probability p clip (y|x), the formula is as follows:

[0063]

[0064] Where d represents the cosine similarity. It is a learnable temperature parameter.

[0065] A feature matching layer is constructed for a few-sample image classification method, and the CoOp model is introduced to optimize the CLIP prompts, generating a matching score tensor and class prototypes. CoOp first defines a set of continuous text vectors V = {v1, v2, v3, v4, v5}. 2, …,v M} as a learnable c, and embed category words into c i The final text embedding is generated by connecting with learnable hints v. The formula is as follows:

[0066]

[0067] By incorporating a small number of samples from downstream tasks, CoOp minimizes the image feature x and the category text embedding. The negative log-likelihood between the two vectors is used to optimize the learnable vector V, as shown in the following formula:

[0068]

[0069] Throughout the process, both the visual encoder and the text encoder remain frozen, and CoOp infers appropriate task-related cues. This is to enhance the model's generalization and discrimination capabilities.

[0070] Step 3: Use an attention mechanism to fuse text features, including the following steps:

[0071] We construct a feature extraction layer in an attention-based text feature fusion method and use this layer to generate initial feature representations. First, we embed z into the input text features. y Perform a linear transformation to generate representations of the query (Q), key (K), and value (V). The formula is as follows:

[0072] Q = W Q ·z y K = W K ·z y V = W V ·z y (17)

[0073] Among them, W Q W K W V These represent the corresponding linear transformation weight matrices.

[0074] Then, the dot product between query Q and key K is calculated, scaled, and an attention score is generated. Normalized attention weights are then generated using the Softmax function, as shown in the following formula:

[0075]

[0076] By performing a weighted summation of normalized attention weights and the value V, a context-dependent feature representation z is generated. f The formula is as follows:

[0077] z f =Attention·V (19)

[0078] Finally, the context features z f Compared with the original text features z yCombined, further feature transformation is performed through a linear transformation layer to generate the final fused feature representation Z. fused The formula is as follows:

[0079] Z fused =W f ·[z f ,z y (20)

[0080] Step four: Design a model optimization method based on the contrastive loss function. First, the text embeddings of manually prompted words and the learnable text embeddings are defined as follows: and The text embeddings of manually prompted words and learnable text embeddings are used as the query (Q), key (K), and value (V) inputs to the attention mechanism module, respectively, to obtain the fused text embedding Z. fused The formula is as follows:

[0081] z fused =Attention(z) coop ,z clip ,z clip ) (twenty one)

[0082] Next, the standard contrast loss L used to fit the base classes is calculated. coop The formula is as follows:

[0083]

[0084] Finally, the contrastive loss L, used to prevent the model from overfitting to the base class, is calculated. attention The formula is as follows:

[0085]

[0086] By combining standard cross-entropy loss and tuning hyperparameters, the model's discriminative ability in both known and unknown tasks can be optimized. The standard cross-entropy loss L... coop Contrast loss L to prevent overfitting attention Combining these, we obtain the final target loss function L, as shown in the following formula:

[0087] L=(1-θ)L coop + θ L attention (twenty four)

[0088] Where θ is a hyperparameter used to balance the effects of the two losses in the final objective task.

[0089] Step 5: Use the obtained attention-based context optimization model to classify images in the image dataset or in the real world;

[0090] This embodiment, through an improved context-optimized prompting algorithm based on an attention mechanism, not only enhances the model's understanding of visual and textual features but also, through diversified prompt generation and the introduction of the attention mechanism, enables the model to exhibit higher robustness and generalization ability when handling complex scenarios. The model's final output is processed using soft labels to ensure the accuracy of the classification results. Ultimately, the optimized CLIP model can provide more accurate classification and recognition results in various application scenarios. The technical effects of this invention are described in detail below with experimental results.

[0091] To verify the classification accuracy of the proposed method in few-shot scenarios, experiments were conducted to compare it with several excellent improved algorithms based on the CLIP model. The experiments were performed on five fine-grained image classification datasets: Flowers102, Stanford Dogs, Fruit92, Veg200, and Heritage, aiming to evaluate the model's classification accuracy at different sample counts and its ability to generalize from the base class to unseen new classes. The experiments were conducted with the following settings: all baseline and improved models were trained in few-shot learning scenarios with 1, 2, 4, 8, and 16 samples, and the test set used the complete dataset for model evaluation. The purpose of the experiments was to compare the classification accuracy of the proposed method with other baseline methods at different sample counts. The training process was based on the CoOp and CLIP models, with implementation details such as using ViT-B / 16 as the visual backbone network and fixing the context text vector length to 4. The template used was "a photo of a[]". To ensure the fairness of the results, all experiments were conducted in the same hardware environment, and the average result of three runs was reported. Figure 3 The classification accuracy of different baseline models on five datasets is presented. Experimental results show that the method described in this embodiment achieves the highest classification accuracy across all sample sizes. The model's performance is particularly significant in 1-shot and 2-shot scenarios. By introducing an attention mechanism to fuse text embeddings before the backpropagation stage, this method not only preserves the classification details of the learnable text embeddings but also captures additional key information from the CLIP model, thereby improving the model's performance in few-shot learning.

[0092] To verify the effectiveness of the introduced attention-based feature fusion module in image classification tasks, this embodiment designed and conducted ablation experiments. The experiments were conducted on the Flowers102, Stanford Dogs, Fruit92, Veg200, and Heritage classification datasets, focusing on testing the model's performance in generalization from base classes to new classes. In this experiment, the performance of models with and without the attention mechanism was compared, and the impact of the attention mechanism on model performance was measured by calculating the difference between three key metrics: base class accuracy, new class accuracy, and harmonic mean H. A positive difference indicates that the metrics after adding the attention mechanism are better than those without, while a negative difference indicates a decline in performance. The experimental results are as follows: Figure 4 As shown in the figure, the results demonstrate that the model with the attention mechanism significantly improves the base class accuracy on all four datasets (Stanford Dogs, Fruit92, Veg200, and Shaanxi Intangible Cultural Heritage Art Image Classification Dataset), particularly in terms of new class accuracy and harmonic mean H, where the attention mechanism significantly enhances the model's generalization ability. Notably, in the Flowers102 dataset, the base class accuracy slightly decreases after adding the attention mechanism. This is primarily because the base classes in this dataset already possess high fitting accuracy, and introducing an additional attention mechanism may, to some extent, disrupt the existing fitting effect. However, even under these circumstances, the addition of the attention mechanism still achieves a comprehensive and significant improvement in new class accuracy and harmonic mean H. This embodiment, through ablation experiments, fully validates the effectiveness of the attention-based feature fusion module in improving the model's generalization ability, providing a solid theoretical basis for further optimization and application of the algorithm.

[0093] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0094] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A contextual cue fine-tuning method based on attention mechanism, characterized in that, The method is specifically designed for few-shot image classification tasks and includes the following steps: Step 1: Utilize the pre-trained visual-language large model CLIP as a foundation. This model learns rich visual and linguistic feature representations by pre-training on a large-scale image-text dataset. Step two introduces a context-optimized prompt strategy. By combining trainable text prompts with category labels, textual knowledge of images in different categories is acquired. Specifically, this includes fine-tuning the CLIP model using learnable prompt words and adapting CLIP to downstream tasks using a visual text encoder with frozen parameters. Step 3: Employ an attention mechanism to fuse text features. An attention network is constructed by designing linear transformations of query (Q), key (K), and value (V), and a Softmax function is used to generate standardized attention weights, enabling effective filtering and integration of key information. Step 4: Design a contrastive loss function, combine it with standard cross-entropy loss, and optimize the model's ability to distinguish between known and unknown tasks by adjusting hyperparameters to balance the recognition performance of different categories. Step 5: Use the obtained attention-based context optimization model to classify images in the image dataset or in the real world.

2. The contextual cue fine-tuning method based on attention mechanism as described in claim 1, characterized in that, Step two further includes: constructing an initial feature extraction layer for the CLIP model, including a visual encoder and a text encoder. The visual encoder maps a given image to a visual embedding space, generating initial visual feature embeddings; the text encoder embeds text into a text embedding space and generates category-related text embeddings through a Transformer encoder. A feature fusion layer for CLIP is constructed, calculating prediction probabilities through similarity matching between visual feature embeddings and text embeddings. A feature matching layer is constructed, and a context optimization (CoOp) model is introduced to optimize the CLIP's prompts, optimizing learnable prompts with a small number of samples to enhance the model's generalization and discriminative abilities.

3. The contextual cue fine-tuning method based on attention mechanism as described in claim 1, characterized in that, Step three further includes: constructing a feature extraction layer in the attention-based text feature fusion method to generate an initial feature representation; performing a linear transformation on the input text feature embedding to generate a representation of query, key, and value; calculating the dot product between the query and key, and generating normalized attention weights using the Softmax function; performing a weighted summation of the normalized attention weights and values ​​to generate a context-dependent feature representation; combining the context features with the original text features, and further generating the final fused feature representation through a linear transformation layer.

4. The contextual cue fine-tuning method based on attention mechanism as described in claim 1, characterized in that, Step four further includes: designing a model optimization method based on a contrastive loss function. The text embeddings of hand-crafted prompts and learnable prompts are used as inputs to the attention mechanism, and the fused text embeddings are calculated. A standard contrastive loss for fitting the base classes and a contrastive loss for preventing model overfitting are calculated.

5. The attention-based contextual cue fine-tuning method as described in any one of claims 1 to 4, characterized in that, The proposed method is applied to few-shot image classification tasks, and enhances the classification accuracy and generalization ability of the model by introducing context optimization and attention mechanisms.

6. A computing device, characterized in that, The computing device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the attention-based contextual cueing fine-tuning method as described in any one of claims 1 to 5.

7. A context-sensitive fine-tuning system based on an attention mechanism, characterized in that, The system includes: a context optimization module for fine-tuning the CLIP model using learnable cue words and acquiring textual knowledge by combining category labels; a feature fusion module for fusing textual features through an attention mechanism to generate the final feature representation; and a model optimization module for optimizing the model's discriminative ability in few-shot image classification tasks through a contrastive loss function.

8. The application of the attention-based contextual cueing fine-tuning method as described in any one of claims 1 to 7 in an image classification device.

Images