Fine-grained multimodal cue learning method based on vision-language pre-training model

Abstract

The present invention discloses a fine-grained multimodal cue learning method based on a visual language pre-training model, which relates to the technical field of visual language pre-training models. The method comprises: constructing a multimodal cue learning vector for learning visual features and text features; generating global differences and subtle visual features and category-differentiating text features between different visual categories based on the fine-grained multimodal cue learning model; and optimizing the multimodal cue learning vector based on a cosine similarity loss function. By combining dual-granularity visual cues with text cues, the present invention effectively solves the problem in existing cue learning and tuning methods that the dual-modal cue representation space cannot be reasoned collaboratively and has difficulty capturing subtle distinguishing features. The method also verifies its superior transfer generalization ability and few-shot learning ability on multiple major image recognition benchmark datasets, achieving significant performance improvements in downstream tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of visual language pre-training models, and in particular to a fine-grained multimodal cue learning method based on a visual language pre-training model. Background Art

[0002] With the development of large-scale vision-language pre-training models (such as CLIP and ALIGN), these models have demonstrated powerful general representation capabilities, ushering in a new paradigm for open-set visual concept recognition. These models are pre-trained on large-scale image-text pairs. The trained models are able to capture a variety of visual concepts from natural language text and transfer the learned representations to a variety of downstream tasks. However, the distribution difference between pre-training data and downstream task data affects the generalization ability of vision-language pre-training models.

[0003] Existing research has shown that cue learning is an effective approach to address this problem. However, designing appropriate cue learning vectors is a difficult task, often requiring extensive domain expertise and time to manually tune the cue words. Furthermore, even after extensive tuning, there is no guarantee that the generated cue will be optimal for downstream tasks. Therefore, how to adapt visual-language pre-trained models to downstream tasks through cue learning, especially in terms of handling class-specific differences and extracting fine-grained visual features, remains an urgent problem.

[0004] In recent years, the application of cue learning in natural language processing (NLP) models has achieved remarkable success, such as text cue tuning, which improves model performance by learning continuously adjustable cue embeddings. Similarly, visual cue tuning introduces visual cues to enhance the ability to learn image representations. However, existing methods mainly focus on unimodal cue tuning and cannot fully represent visual concepts, especially in multimodal cue learning, where visual and textual concepts and their alignment require deeper reasoning. In addition, existing methods often only utilize global visual features and ignore the local fine-grained information of the target object, resulting in poor performance of the model on tasks with insignificant local difference information. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies of the existing technology and to solve the problem in the existing technology that it is impossible to effectively take into account the global high-level semantics and fine-grained specific features of the visual modality and textual modality when processing multimodal tasks. The present invention can improve the learning ability of visual and textual cues by introducing dual-granularity visual cues, class-specific text cues, multimodal cues generation modules, and multimodal cues tuning mechanisms, especially in the extraction of fine-grained class-specific features. It specifically provides a "technical solution that can freeze the text encoder and visual encoder of the visual language pre-training model when migrating and processing visual and textual downstream tasks, and only optimize the parameters of the visual cues and text cues to achieve the extraction of global and fine-grained visual information, as well as the class-specific description of the text, ensure the alignment and collaborative reasoning between multimodal features, and improve the generalization performance of the model in downstream tasks", as follows:

[0006] 1) In the first aspect, the present invention provides a fine-grained multimodal cue learning method based on a visual language pre-training model. The specific technical solution is as follows:

[0007] S1. Construct a multimodal cue learning vector for learning visual and text features:

[0008] S101. Construct a fine-grained multimodal cue learning model based on a visual language pre-training model. The fine-grained multimodal cue learning model includes visual cues and text cues. The fine-grained multimodal cue learning model is used to learn visual cues and text cues. Specifically, the image encoder and text encoder of the visual language pre-training model are used to process the input image and K category labels to generate a learnable visual cue V. i and learnable text prompts T i , where the input image is denoted as K is the total number of category labels, H is the height of the input image, W is the width of the input image, C is the channel of the input image, and i is the network layer index;

[0009] S102, Learnable Visual Cues V i Designing global visual cues for global high-level semantic information and fine-grained visual cues for extracting subtle differences between categories. The global visual cues are used to extract global semantic information of the input image, and the fine-grained visual cues are used to extract subtle differences between visual categories.

[0010] S103, Learnable Text Prompt T i Adopting category label-aware cue learning, the category label is represented by an embedded vector, and the category label is used as the input of the learnable text cue to generate category-differentiated text features;

[0011] S104. Generating global visual cues, fine-grained visual cues, and learnable text cues in a shared latent semantic space so that the fine-grained multimodal cue learning model can achieve cross-modal information interaction and alignment, wherein the multimodal cue learning vector includes the global visual cues, the fine-grained visual cues, and the learnable text cues;

[0012] S2. Generate visual features and category-differentiated text features based on the fine-grained multimodal cue learning model:

[0013] S201. Generate category-specific text features through a text encoder:

[0014]

[0015] In formula (1), w represents the embedding vector of the category label, T b Represents text prompts, category-differentiated text features Z k Generated by the text encoder L(·);

[0016] S202: Generate global visual features and fine-grained features of the input image through a visual encoder:

[0017]

[0018] In formula (2), e represents the image block embedding, V b Represents visual cues, features X include fusion features of global visual features and fine-grained features, and features X are generated by the image encoder I(·);

[0019] S3. Optimize the multimodal cue learning vector based on the cosine similarity loss function:

[0020] The learnable visual cue and the learnable text cue are optimized based on the cosine similarity loss function, and the similarity difference between the visual features and the category-differentiated text features is calculated by minimizing the similarity difference between the visual features and the category-differentiated text features. The probability of the k-th category label is calculated by formula (3)

[0021]

[0022] Among them, τ is a fixed hyperparameter, sim(x,Z k ) is the similarity between visual features and category-differentiating text features.

[0023] The beneficial effects of the fine-grained multimodal cue learning method based on the visual language pre-training model provided by the present invention are as follows:

[0024] Learnable global and fine-grained visual cues are used to generate global and fine-grained visual features of the input image. Learnable textual cues process the category labels of the input image to generate category-specific textual features of the input image. At the same time, visual and textual cues are generated based on a shared latent semantic space to promote cross-modal interaction in the fine-grained multimodal cue learning model. Furthermore, the similarity between the category-specific text features of the input image and the global and fine-grained visual fusion features is calculated and based on the similarity, the visual and textual cues are semantically aligned and reinforce each other, promoting the cross-modal collaborative training and reasoning of the fine-grained multimodal cue learning model to achieve the optimization of the multimodal cue learning vector and the training of the visual language pre-training model. The present invention effectively solves the problem that the dual-modal cue representation space in the existing cue learning and tuning methods cannot be collaboratively reasoned and is difficult to capture subtle distinguishing features by combining dual-granularity visual cues (global visual cues and fine-grained visual cues) with text cues. Its superior transfer generalization ability and few-sample learning ability have been verified on multiple major image recognition benchmark datasets, and significant performance improvement has been achieved in downstream tasks (multimodal tasks).

[0025] Based on the above solution, the fine-grained multimodal prompt learning method based on the visual language pre-training model of the present invention can also be improved as follows.

[0026] Furthermore, in S102, the process of obtaining the global visual features includes:

[0027] The global visual cue branch extracts the global visual features of the input image through formula (4):

[0028] [G i ,C i ,E i ]=I i ([G i-1 ,C i-1 ,E i-1 ]) (4)

[0029] Where i is the layer index, I i represents the i-th visual encoder layer, C i and E i Represent the [CLASS] embedding and image patch embedding of each image respectively.

[0030] Furthermore, in S102, the process of obtaining fine-grained visual cues includes:

[0031] The fine-grained visual cue branch sets the activation feature map A of the input image c ∈R H×W , where H and W represent the height and width of the input image respectively, the image block size is P, and the number of image blocks is: The set P of all image patches in the activation feature map c for:

[0032]

[0033] in, Represents the block at row m and column n on the activation feature map;

[0034] Based on the set mask matrix M c , perform random mask operations on the image blocks in the activation feature map, M c ∈R H×W , and set the mask matrix M c for:

[0035]

[0036] Among them, Bernoulli distribution represents the probability distribution of mask operation, P patch represents the probability of masking the image block, M c The mask of the highest activation value position in (i, j) is set to 0, which is specifically expressed by formula (7):

[0037] M c (i,j)=0,ifA c (i,j)=max(A c ) (7)

[0038] Through the activated feature map A′ after suppression c Generate fine-grained visual cues F, F = {F j ∈R d ∣∣j∈N,1≤j≤n f}, specifically:

[0039]

[0040] Where λ represents the inhibition factor;

[0041] Fine-grained visual cues are input to the fine-grained vision branch via formula (9):

[0042] [F i ,C i ,E′ i ]=I i ([F i-1 ,C i-1 ,E′ i-1 ]) (9)

[0043] Among them, E i ′ represents the image patch embedding after suppression.

[0044] Furthermore, in S103, class-aware text prompts are designed to adapt to downstream tasks by automatically generating text prompts that semantically match the visual class-specific concepts. This is specifically accomplished through formula (10):

[0045] [_,W i ]=L i ([T i-1 ,W i-1 ]) (10)

[0046] Among them, the text prompt of the k-th category label is T k ={u1,…,u M ,c i},{u1,…,u M} represents the learnable text context vector, and the word embedding vector of the text class tag is c i , L i is the i-th layer of the text encoder, W i is the word embedding representation.

[0047] Furthermore, in S104, generating the global visual cue, the fine-grained visual cue, and the learnable text cue in the shared latent semantic space further includes:

[0048] A multimodal cue generation module is proposed to map global and fine-grained visual cues in a shared semantic space to each layer of the visual encoder through a linear layer, which is specifically accomplished by formula (11):

[0049]

[0050] Where i is the layer index, N is the total number of layers, and the length of the text prompt, global visual prompt and fine-grained visual prompt of each layer is the same. A linear projection layer representing global visual cues, Linear projection layer for representing fine-grained visual cues.

[0051] Furthermore, in S3, the calculation is performed by minimizing the similarity difference between the visual feature and the category-differentiated text feature, including:

[0052] Design a multimodal cue tuning evaluation method, using visual embedding and class-aware text embedding of global and fine-grained visual cue branches, setting y to be the true label, and optimizing the visual classification loss Enables fine-grained multimodal cue learning models to find subtle differences between related categories and optimize visual classification losses for:

[0053]

[0054] Assume that the category-specific visual embedding and text embedding are E G and E T , using formula (13) to calculate the category-specific global difference prediction, namely the global difference loss To align the visual embeddings with the category-specific text embeddings:

[0055]

[0056] Among them, E G represents the global visual embedding, represents the text embedding corresponding to category i, and τ is a hyperparameter that controls the sharpness of the distribution;

[0057] The visual embedding and text embedding pair of the fine-grained visual cue branch is represented as (E F ,E T ), calculate the fine-grained difference loss The calculation formula is:

[0058]

[0059] in, represents the text embedding of the corresponding category i, τ is a hyperparameter that controls the sharpness of the distribution, and is supervised by To enhance the ability to extract subtle information features;

[0060] Will and For optimization, the overall loss function formula is:

[0061]

[0062] 2) In a second aspect, the present invention further provides a system for implementing any of the above-mentioned methods for fine-grained multimodal cue learning based on a visual language pre-training model. The specific technical solution is as follows:

[0063] It includes: dual-granularity visual prompt module, text prompt module, visual text prompt alignment generation module and visual text prompt tuning module;

[0064] The dual-granularity visual cue module is used to provide: global visual cues for global high-level semantic information and fine-grained visual cues for extracting subtle differences between categories. The global visual cues are used to extract global semantic information of the input image, and the fine-grained visual cues are used to extract subtle differences between visual categories.

[0065] The text prompt module is used to: generate text prompts that are aware of category labels;

[0066] The visual-textual prompt alignment generation module is used to: perform cross-modal alignment and collaborative reasoning of visual and textual prompts to generate consistent semantic representations. The visual prompts include global visual prompts and fine-grained visual prompts.

[0067] The visual-textual cue tuning module is used to optimize the performance of visual cues and textual cues in downstream tasks based on loss functions of different dimensions.

[0068] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor coupled to a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor, so that the electronic device implements any of the above-mentioned fine-grained multimodal prompt learning methods based on a visual language pre-training model.

[0069] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned fine-grained multimodal prompt learning methods based on a visual language pre-training model.

[0070] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments of the present invention:

[0072] Figure 1 Schematic diagram of a flow chart of a fine-grained multimodal cue learning method based on a visual language pre-training model according to an embodiment of the present invention;

[0073] Figure 2 Schematic diagram of the structure of the fine-grained multimodal cue learning model;

[0074] Figure 3 is the relationship between the hyperparameter weights α, β, and γ and the classification accuracy. DETAILED DESCRIPTION

[0075] The principles and features of the present invention are described below. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0076] The following describes in detail the technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems using specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The following embodiments of the present invention are described in conjunction with the accompanying drawings.

[0077] Explanation of the English abbreviations involved in this application:

[0078] 1) CLIP is the abbreviation of Contrastive Language-Image Pretraining, which means contrastive language-image pretraining model.

[0079] 2)MPT is the abbreviation of Multimodal Prompt Tuning.

[0080] 3) GVP is the abbreviation of Global Visual Prompt, global visual prompt.

[0081] 4) FG-VP is the abbreviation of Fine-grained Visual Prompt.

[0082] 5) CATP is the abbreviation of Class-Aware Textual Prompt, which means class-aware textual prompt.

[0083] like Figure 1 As shown, a fine-grained multimodal cue learning method based on a visual language pre-training model according to an embodiment of the present invention includes the following steps:

[0084] S1. Construct a multimodal cue learning vector for learning visual and text features:

[0085] S101. Construct a fine-grained multimodal cue learning model based on a visual language pre-training model. The fine-grained multimodal cue learning model includes visual cues and text cues. The fine-grained multimodal cue learning model is used to learn visual cues and text cues. Specifically, the image encoder and text encoder of the visual language pre-training model are used to process the input image and K category labels to generate a learnable visual cue V. i and learnable text prompts T i , where the input image is denoted as K is a positive integer, H represents the height of the input image, W represents the width of the input image, C represents the channel of the input image, i represents the network layer index, and K is the total number of category labels;

[0086] S102, Learnable Visual Cues V iWe design global visual cues for global high-level semantic information and fine-grained visual cues for extracting subtle differences between categories. Global visual cues are used to extract global semantic information of the input image, while fine-grained visual cues are used to extract subtle differences between categories.

[0087] S103, Learnable Text Prompt T i Adopting category label-aware cue learning, the category label is represented by an embedded vector, and the category label is used as the input of the learnable text cue to generate category-differentiated text features;

[0088] S104. Generating global visual cues, fine-grained visual cues, and learnable text cues in a shared latent semantic space so that the fine-grained multimodal cue learning model can achieve cross-modal information interaction and alignment, wherein the multimodal cue learning vector includes the global visual cues, the fine-grained visual cues, and the learnable text cues;

[0089] S2. Generate visual features and category-differentiated text features based on the fine-grained multimodal cue learning model:

[0090] S201. Generate category-specific text features through a text encoder:

[0091]

[0092] In formula (1), w represents the embedding vector of the category label, T b Represents text prompts, category-differentiated text features Z k Generated by the text encoder L(·);

[0093] S202: Generate global visual features and fine-grained features of the input image through a visual encoder:

[0094]

[0095] In formula (2), e represents the image block embedding, V b Represents visual cues, features X include fusion features of global visual features and fine-grained features, and features X are generated by the image encoder I(·);

[0096] S3. Optimize the multimodal cue learning vector based on the cosine similarity loss function:

[0097] Based on the cosine similarity loss function, the learnable visual cues and the learnable text cues are optimized by minimizing the similarity difference between the visual features and the category-differentiated text features. The probability of the k-th category label is calculated by formula (3)

[0098]

[0099] Among them, τ is a fixed hyperparameter, sim(x,Z k ) is the similarity between visual features and category-differentiated text features, 0≤k≤K.

[0100] Optionally, in S102, a global visual prompt is performed based on the multimodal information of the input text and image [G i ,C i ,E i ], the process of obtaining global visual features includes:

[0101] The global visual cue branch extracts the global visual features of the input image through formula (4):

[0102] [G i ,C i ,E i ]=I i ([G i-1 ,C i-1 ,E i-1 ]) (4)

[0103] Where i is the layer index, I i represents the i-th visual encoder layer, C i and E i Represent the [CLASS] embedding and image patch embedding of each image respectively.

[0104] Optionally, in S102, visual cues for global high-level semantic information and fine-grained visual cues for extracting subtle differences between categories are designed to obtain fine-grained visual cues [F i ,C i ,E i ′], the process of obtaining fine-grained visual cues includes:

[0105] The fine-grained visual cue branch sets the activation feature map A of the input image c ∈R H×W , where H and W represent the height and width of the input image respectively, the image block size is P, and the number of image blocks is: The set P of all image patches in the activation feature map c for:

[0106]

[0107] in, Represents the block at row m and column n on the activation feature map;

[0108] Based on the set mask matrix M c , perform random mask operations on the image blocks in the activation feature map, Mc ∈R H×W , and set the mask matrix M c for:

[0109]

[0110] Among them, Bernoulli distribution represents the probability distribution of mask operation, P patch represents the probability of masking the image block, M c The mask of the highest activation value position in (i, j) is set to 0, which is specifically expressed by formula (7):

[0111] M c (i,j)=0,ifA c (i,j)=max(A c ) (7)

[0112] Through the activated feature map A′ after suppression c Generate fine-grained visual cues F, F = {F j ∈R d ∣∣j∈N,1≤j≤n f}, specifically:

[0113]

[0114] Among them, λ represents the suppression factor. In the experiment, λ is set to 0.1. It can also be set according to actual conditions. The suppression factor is used to adjust the value of the suppressed feature.

[0115] Fine-grained visual cues are input to the fine-grained vision branch via formula (9):

[0116] [F i ,C i ,E′ i ]=I i ([F i-1 ,C i-1 ,E′ i-1 ]) (9)

[0117] Among them, E′ i Represents the suppressed image patch embedding.

[0118] It should be noted that the fine-grained visual cue branch improves feature representation capabilities by suppressing salient regional features in the visual activation map.

[0119] Optionally, in S103, the design-type perception text prompt includes:

[0120] Adapting to downstream tasks is achieved by automatically generating textual prompts that semantically match visual class-specific concepts, as shown in formula (10):

[0121] [_,W i ]=L i ([T i-1 ,W i-1 ]) (10)

[0122] Among them, the text prompt of the k-th category label is T k ={u1,…,u M ,c i},{u1,…,u M} represents the learnable text context vector, and the word embedding vector of the text class tag is c i , L i is the i-th layer of the text encoder, W i is the word embedding representation.

[0123] Optionally, in S104, generating global visual cues, fine-grained visual cues, and learnable text cues in the shared latent semantic space further includes:

[0124] A multimodal cue generation module is proposed to map global and fine-grained visual cues in a shared semantic space to each layer of the visual encoder through a linear layer, which is specifically accomplished by formula (11):

[0125]

[0126] Where i is the layer index, N is the total number of layers, which is N=12 in this framework. The length of textual cues, global visual cues, and fine-grained visual cues in each layer is the same. A linear projection layer representing global visual cues, Linear projection layer for representing fine-grained visual cues.

[0127] Optionally, in S3, the calculation is performed by minimizing the similarity difference between the visual features and the category-differentiated text features, including:

[0128] We design a multimodal cue tuning evaluation method that uses visual embeddings and class-aware text embeddings from global and fine-grained visual cue branches. Visual classification loss is used to encourage the model to find subtle differences between similar categories, thereby enabling the fine-grained multimodal cue learning model to extract more discriminative features. Specifically:

[0129] Set y to be the true label, and optimize the visual classification loss Enables fine-grained multimodal cue learning models to find subtle differences between related categories and optimize visual classification losses for:

[0130]

[0131] Assume that the category-specific visual embedding and text embedding are E G and E T , using formula (13) to calculate the category-specific global difference prediction, namely the global difference loss To align the visual embeddings with the category-specific text embeddings:

[0132]

[0133] Among them, E G represents the global visual embedding, represents the text embedding corresponding to category i, τ is a hyperparameter that controls the sharpness of the distribution, and is used to optimize the visual classification loss. and fine-grained difference loss It can explicitly promote the vision-language pre-training model to extract fine-grained category differences, thereby enhancing the model's discriminative ability.

[0134] In order to improve the acquisition of category-specific differences and enhance the ability to extract subtle information features, a loss for predicting fine-grained differences is designed. The fine-grained region prediction function is used to capture more informative features of a specific category. Specifically, the visual embedding and text embedding pair of the fine-grained visual cue branch is represented as (E F ,E T ), calculate the fine-grained difference loss The calculation formula is:

[0135]

[0136] in, represents the text embedding of the corresponding category i, τ is a hyperparameter that controls the sharpness of the distribution, and is supervised by To enhance the ability to extract subtle information features;

[0137] Category-specific global visual cues and category-aware textual cues will gradually align, thus improving the ability of visually trained models to acquire category-specific features. The subtle differences between visual categories will be ignored. Diverse category-related cues can be integrated to recover the missing differential features in formula (13) in category-specific visual and textual cues, thereby ensuring that the pre-trained CLIP model can perceive more comprehensive differential features and improve the model's discriminative ability.

[0138] Will and For optimization, the overall loss function formula is:

[0139]

[0140] Among them, α is The weight hyperparameter, β is The weight hyperparameter, γ is The weight hyperparameters.

[0141] Among them, the visual language pre-training model can be a CLIP model or an ALIGN model, and can also be set according to actual conditions.

[0142] Example 1:

[0143] The purpose of this invention is to provide a fine-grained multimodal prompt learning method based on a visual language pre-training model. Specifically, a visual and text joint learning framework for fine-grained multimodal prompt learning (FGMPL) is proposed, namely a fine-grained multimodal prompt learning model based on a visual language pre-training model (also known as a multimodal prompt learning framework). The multimodal prompt learning framework generates and optimizes visual prompts and text prompts. By combining visual prompts and text prompts, the model's global and fine-grained feature extraction capabilities in multimodal tasks are improved. The implementation process of the fine-grained multimodal prompt learning method based on a visual language pre-training model is as follows:

[0144] 1) Input data preparation: The model accepts input data, mainly including image data (I∈R W×H×C ) and the corresponding category labels, there are K category labels in total. The image data is processed by the image encoder of the visual language pre-training model, while the category labels are processed by the text encoder to generate preliminary visual cues and text cues, that is, the learnable visual cues V i and learnable text prompts T i .

[0145] 2) Multimodal prompt generation:

[0146] ① Visual Cue Generation: Through a multimodal cue learning framework, two types of visual cues are generated: global visual cues and fine-grained visual cues. Global visual cues are used to capture high-level semantic information of the image, while fine-grained visual cues focus on identifying subtle differences between categories.

[0147] ②Textual prompt generation: Textual prompts are generated based on category labels and optimized in a category-specific textual prompt module to form textual prompts that are coordinated with visual prompts.

[0148] ③ Cue alignment and feature fusion: Through a shared latent semantic space, visual cues and textual cues are aligned at the semantic level. The combination of global cues and fine-grained visual cues ensures the effective fusion of global information and category-specific information of the image.

[0149] ④ Loss function training optimization: By optimizing the cosine similarity loss function, the matching performance of image and text features is further improved.

[0150] ⑤ Global and fine-grained difference loss: Global difference loss and fine-grained difference loss are designed to improve the alignment performance of global cues and fine-grained visual cues respectively, ensuring the effective capture of subtle differences between classes.

[0151] ⑥ Cue Tuning and Training: Throughout the training process, the encoder of the vision-language pre-trained model remains frozen, and only the parameters of the visual and textual cues are tuned. By optimizing the cue learning vectors, efficient reasoning and alignment of visual and textual cues are ultimately achieved.

[0152] ⑦ Output results: Based on the optimized prompt learning, classification results are generated, which improves the accuracy of image classification tasks, especially in fine-grained image classification tasks.

[0153] This process ensures efficient collaboration between visual and textual cues in multimodal tasks, enabling the model to accurately identify fine-grained information in complex images and accurately match it with textual information. The specific implementation process of Example 1 is as follows:

[0154] 1) Multimodal prompt generation and feature extraction.

[0155] Use the multimodal cue adjustment framework and CLIP model for cue generation and feature extraction. Process the input image and category labels through visual cues and textual cues to generate visual cues and textual cues;

[0156] Through the multimodal prompt adjustment framework, visual prompts and text prompts are learned and adjusted. The image encoder of the vision-language pre-training model processes the input image I∈R W×H×C And the text encoder processes K category labels to generate learnable visual cues: And the text prompt:

[0157] Visual cues and textual cues are generated through a shared latent semantic space. i and text prompt T i By generating a shared latent semantic space, cross-modal interaction is promoted, supporting the extraction of global high-level semantic information and fine-grained feature learning between categories. The final representation of image and text features is calculated using the formula. The image features and text features of the output image are represented by formulas (1) and (2).

[0158] Model training optimization is performed based on cosine similarity. During training, cosine similarity is used to optimize the performance of extracting image features and text features from visual cues and text cues, and the prediction probability of category k is calculated using formula (3).

[0159] It should be noted that the visual cue V i and text prompt T i When performing joint tuning, the encoders (text encoder and visual encoder) are kept frozen and the multimodal cue learning vector is optimized so that V (the optimized visual cue V i ) and T(optimized text prompt T i ) to suit specific mission requirements.

[0160] 2) Design of dual-granularity visual cues and class-aware text cues:

[0161] Design global and fine-grained visual cues. This dual-grained cue design leverages global visual cues to capture high-level global semantic information, while fine-grained visual cues extract category-specific subtle features. These two cues capture different levels of image features, enhancing the model's ability to extract features.

[0162] 3) Feature extraction of global visual cues.

[0163] Global visual cues extract global high-level semantic information, which is achieved through formula (4).

[0164] 4) Design of fine-grained visual cues.

[0165] Fine-grained visual cues improve feature representation capabilities by suppressing salient regional features, setting the activation feature map A c ∈R H×W , where H and W represent the height and width of the image respectively. The image block size is P and the number of image blocks is The random mask operation is used to improve the fine-grained feature extraction effect. The fine-grained visual cue generation formula is formula (8), and the fine-grained visual cue is input into the visual cue branch through formula (9).

[0166] 5) Design of class-aware text prompts:

[0167] Class-aware textual cues adapt to different tasks by automatically generating textual cues that match the specific semantics of visual classes. This is accomplished through formula (10).

[0168] 6) Multimodal prompt generation module:

[0169] The multimodal cue generation module maps global visual cues, fine-grained visual cues, and textual cues to each layer of the visual encoder through a linear layer. The specific formula is formula (11).

[0170] 7) Multimodal prompt tuning evaluation method, a multimodal prompt tuning evaluation method is designed through the joint learning of visual and textual prompts. Fine-grained visual classification loss is optimized by cross entropy loss. Optimize and optimize visual classification loss The calculation process is formula (12).

[0171] Category-specific visual text alignment is used to perform global difference prediction using formula (13) to obtain the global difference loss By optimizing The model can be explicitly forced to find fine-grained category differences, thereby enhancing the model's discriminative ability.

[0172] In order to improve the acquisition of category-specific differences and enhance the ability to extract subtle information features, a fine-grained region prediction function is designed. To capture more subtle feature information of a specific category. Specifically, the visual embedding and text embedding pair of the fine-grained visual cue branch is represented as (E F ,E T ). The discriminative ability of fine-grained regions is further improved by the fine-grained region prediction loss, i.e., the fine-grained difference loss To achieve, fine-grained difference loss Calculated by formula (14). Category-specific global visual cues and category-aware textual cues will gradually align, thereby improving the ability of the pre-trained CLIP model to obtain category-specific features. Will ignore the subtle differences between visual categories, so the final loss Specifically:

[0173] right Global discrepancy loss and fine-grained region prediction loss After optimization, the overall loss function formula is formula (15).

[0174] Example 2:

[0175] In Example 2, a fine-grained multimodal cue learning model based on the CLIP model is constructed, and the network structure is as follows: Figure 2 As shown, Figure 2 The symbols appearing in are explained as follows:

[0176] 1) Flame symbol: This symbol appears in multiple modules, indicating that the module's parameters are trainable. Flame symbols are commonly found in the cue generation module, the fine-grained visual cue module, and the Transformer layer. The parameters of these parts are optimized according to the requirements of the downstream task during training.

[0177] 2) Snowflake symbols next to the text encoder and visual encoder: Snowflake symbols represent frozen parameters, which are usually used to indicate that the parameters of these parts will not be updated in downstream tasks. Figure 2 Both the text encoder and the visual encoder have snowflake symbols, which means that their parameters are obtained from the pre-trained model. They will not be trained when performing downstream tasks and are only responsible for feature extraction.

[0178] 3) Visual cues V i :exist Figure 2 In the visual cue, the symbol V is used i Represents, where i indicates that the visual cue comes from the i-th layer of the visual encoder. Visual cue V i It contains global and fine-grained information extracted from visual data, and after processing through the visual encoder, it can generate hints for image features. The final visual hints are input into the multimodal hint generation module for fusion.

[0179] 4) Text prompt T i :Text prompts use the symbol T i Represented as , where i indicates that the textual prompt comes from the i-th layer of the text encoder. Textual prompts combine class labels and context vectors to provide class-specific descriptions of text features. Like visual prompts, textual prompts are also input into the multimodal prompt generation module.

[0180] 5) Global visual cues: ({G j ∈R d ∣∣j∈N,1≤j≤n g}) is a symbolic representation of the global visual cue, where G j Represents the global cue vector extracted during the visual encoding process, d represents the dimension of the vector, n g Represents the length of global hints. These hints capture the global visual feature information of the input image and are generated in different layers.

[0181] 6) Image block embedding E i :This symbol stands for image block embedding, which is the feature embedding extracted from the image in the visual encoder. During the image encoding process, the image is divided into multiple blocks, and each block generates an embedding vector (E i ), these vectors are collectively used to generate visual cues.

[0182] 7) Vector space representation R d : (R d ) represents a real vector space of dimension (d). For example, both visual cues and textual cues exist in (R d) space. During the processing of visual and textual cues, all cue vectors are ultimately mapped into the same dimensional space, facilitating interaction in the multimodal cue generation module.

[0183] 8) Similarity Calculation: In the process of generating text and visual cues, sim(·) represents the similarity calculation, which is usually used to calculate the cosine similarity between two vectors. Similarity calculation is used to determine the degree of match between visual cues and text cues in semantic space.

[0184] 9) Temperature hyperparameter τ: τ is Figure 2 The temperature hyperparameter used for similarity calculations in . It is used to adjust the smoothness of the similarity distribution when calculating the match between the visual and textual cues. Generally, higher values ​​of τ make the distribution flatter, while lower values ​​make the distribution sharper.

[0185] 10) Mask matrix M c : It is a mask matrix in fine-grained visual cue generation, used to suppress certain areas in the activation feature map. (M c ) is represented in binary form, 1 represents the masked area and 0 represents the unmasked area. The use of this mask encourages the model to extract more fine-grained information from the unsuppressed area.

[0186] 11) Suppression coefficient λ: λ is used to adjust the value of suppressed features in the fine-grained visual cue branch. Typically, it is set to a low value (e.g., 0.1) to suppress features embedded in certain image patches, helping the model extract information from other detail areas.

[0187] Figure 2 The formula symbols involved are mainly used to describe the extraction, transformation and interaction of visual and text features in the prompt generation process. i 、T i , sim(·), τ, etc., clearly demonstrate the process of multimodal cue generation and alignment. These symbols effectively match and optimize between visual and textual features, enabling the model to better handle downstream tasks.

[0188] Example 2 proposes a joint learning framework for visual and textual learning of fine-grained multimodal prompts. This framework is based on the fine-grained multimodal prompt learning method of the CLIP model. It generates and optimizes visual prompts and textual prompts through the multimodal prompt learning framework, aiming to improve the ability to extract visual and textual features of the pre-trained model. The implementation process is as follows:

[0189] First, the CLIP model's image and text encoders process the input image and category labels to generate two types of cues: global visual cues and fine-grained visual cues. The global visual cue branch extracts global semantic information from the image, while the fine-grained visual cue branch focuses on capturing subtle differences between categories, such as local features and texture details. Simultaneously, class-specific textual cues are generated based on the category labels and aligned with the visual cues in a shared latent semantic space. To ensure effective cross-modal feature fusion, the system measures the similarity between image and text features using cosine similarity, and then optimizes the learned cue vector using a cosine similarity loss function. To further improve matching performance, the method designs a global difference loss and a fine-grained difference loss function to enhance the alignment of global and fine-grained visual cues, respectively. Throughout this process, the CLIP model's encoder remains frozen, and only the cue parameters are optimized and trained. Through the collaborative learning and fine-tuning of visual and textual cues, the model effectively improves the accuracy of cross-modal matching between images and text, demonstrating particularly strong performance improvements in fine-grained image classification tasks, ensuring higher classification accuracy and cross-modal understanding. The specific steps involved are as follows:

[0190] 1) Multimodal prompt learning feature extraction module:

[0191] This example demonstrates a process for optimizing and generating multimodal cue learning vectors, primarily for learning a joint representation of visual and textual features. The FGMPL framework includes a dual-granularity visual cue branch, class-specific textual cue generation, and a multimodal cue generation and tuning module. During downstream task training, the encoder is frozen, and only parameters related to cue learning are trainable.

[0192] In Example 2, CLIP is selected as the backbone model of the framework. The CLIP model includes an image encoder and a language encoder. W×H×C and K category labels to generate visual cues and text prompts The image features and text features of the input image are jointly expressed as formula (1) and formula (2). The final image features and text features output by the CLIP model can be measured by calculating the cosine similarity between them, which is specifically achieved by formula (3). The probability of each category k can be calculated by formula (3): Used to measure the possibility that the input image belongs to category k.

[0193] 2) Dual-granularity visual cues: The visual cue module consists of a global visual cue branch and a fine-grained visual cue branch, which are used to extract global and class-specific fine-grained features, respectively.

[0194] ① Global visual cue branch:

[0195] The global visual cue branch aims to capture the global discriminative information of the input image. By generating global cue vectors at different levels of the visual encoder, the global visual features of the image are captured layer by layer. The global visual cue generation process is expressed using Equation (4). Through this global visual cue, the model is able to capture the global information of the image from different levels.

[0196] ② Fine-grained visual cue branch:

[0197] The goal of the fine-grained visual cue branch is to discover more subtle class-specific features. By designing a mask mechanism to suppress salient areas, the network is prompted to focus on other fine-grained information areas. First, define the class activation map A c ∈R H×W , and generate the mask matrix M through formula (5) c , in order to generate a random mask M c , use Bernoulli distribution to sample the activation patches, specifically obtained by formula (6), then use the mask to suppress features outside the peak position, and adjust the suppressed area according to formula (7) to obtain formula (8).

[0198] 3) Class-specific textual hint generation:

[0199] To further enhance the alignment of visual cues and textual cues, this embodiment proposes a class-specific textual cues generation module. This module uses learnable context vectors and word embeddings to generate textual cues that are aligned with visual cues. Let {u1,…,u M} is the learnable text context vector, c i is a class-specific word embedding, then the text hint T of the kth class k Indicates T k ={u1,…,u M ,c i}, the text prompt is embedded layer by layer through the text encoder, and formula (10) is obtained.

[0200] 4) Multimodal prompt generation:

[0201] The multimodal cue generation module is used to generate visual cues and textual cues in a shared semantic space and ensure their alignment. Visual cues and textual cues are mapped via a linear projection layer as shown in Equation (11).

[0202] 5) Multimodal prompt tuning:

[0203] In order to improve the alignment and discrimination of multimodal prompts, a multimodal prompt tuning module is designed. The alignment of visual and textual prompts can be achieved through the following three loss functions: visual classification loss Class-specific global discrepancy loss and fine-grained difference loss The calculation formula of visual classification loss is formula (12), global difference loss The calculation formula is formula (13), the fine-grained difference loss The calculation formula is formula (14). The overall loss function of the FGMPL framework combines the alignment loss of visual cues and textual cues and the class-specific prediction loss. The overall loss function is formula (15). By optimizing the overall loss function, the performance of FGMPL in multimodal cue learning can be significantly improved.

[0204] The present invention was subjected to the following comparative experiment:

[0205] The effectiveness of the proposed method was verified on 11 widely used datasets. Experimental results show that compared with other cue tuning methods, the fine-grained multimodal cue learning framework exhibits significant advantages in multiple downstream tasks, especially when dealing with tasks that require distinguishing fine-grained information. The experimental results are verified and analyzed as follows:

[0206] 1. Experimental setup

[0207] a. The experimental task is set as follows:

[0208] The technical solution of the present invention verifies the proposed method in the following challenging task settings:

[0209] ① Generalization from basic categories to new categories:

[0210] The dataset is divided into base categories and novel categories. A fine-grained multimodal cue learning model based on a pre-trained vision-language model is trained using a very small number of images in the base categories, and its performance is evaluated on unseen novel categories to assess the model's zero-shot generalization ability.

[0211] ② Cross-dataset evaluation: After training on ImageNet, the fine-grained multimodal cue learning model based on the visual language pre-training model is directly evaluated on multiple other datasets. To compare with previous work, the model is trained on 1000 categories of ImageNet with a small sample size.

[0212] ③ Domain Generalization: We evaluate the robustness of our proposed method on out-of-distribution datasets. Using a model trained on ImageNet, we directly evaluate it on four ImageNet variants containing different types of offsets.

[0213] b. Dataset settings:

[0214] To generalize from base categories to new categories and evaluate across datasets, we evaluated the performance of our method on 11 image classification datasets, covering a wide range of recognition tasks. These datasets include two common image recognition datasets: ImageNet and Caltech101; five fine-grained image classification datasets: StanfordCars, OxfordPets, Food101, Flowers102, and FGVCAircraft; the scene recognition dataset SUN397; the action recognition dataset UCF101; the texture dataset DTD; and the satellite imagery dataset EuroSAT. For domain generalization, we used ImageNet as the source dataset, and ImageNet-V2, ImageNet-Sketch, ImageNet-A, and ImageNet-R as the target datasets.

[0215] The following is a detailed introduction to the mentioned datasets:

[0216] 1. Common object datasets:

[0217] 1.1 ImageNet:

[0218] ImageNet is a large-scale visual dataset containing over 14 million annotated images with over 21,000 class labels. Approximately 1,000 of these class labels are widely used for image classification tasks. The ImageNet dataset plays a crucial role in computer vision, particularly in tasks such as image classification, object detection, and semantic segmentation, becoming a standard benchmark for evaluating model performance.

[0219] 1.2Caltech101:

[0220] Caltech101 is an image classification dataset covering 101 object categories, with each category containing approximately 40 to 800 images. The images are high-resolution and clearly positioned. The main challenges of this dataset lie in its uneven category distribution and relatively small number of samples, making it suitable for tasks such as image classification and object detection.

[0221] 2. Fine-grained datasets:

[0222] 2.1StanfordCars:

[0223] StanfordCars is a fine-grained image classification dataset focused on vehicle recognition. The dataset contains 16,185 vehicle images classified into 196 different vehicle model categories. The images contain significant variations in perspective, lighting, and background, making it suitable for studying fine-grained classification problems.

[0224] 2.2OxfordPets:

[0225] OxfordPets is a pet identification dataset containing 7,349 annotated images of 37 cat and dog breeds. The dataset is challenging due to the subtle differences between different breeds, making it suitable for tasks such as pet species classification and object detection.

[0226] 2.3Food101:

[0227] Food101 is an image dataset for food classification, containing 101 different food categories, with 1,000 images per category. This dataset is widely used in food recognition and image classification tasks, which are particularly challenging for distinguishing between highly diverse and similar categories.

[0228] 2.4Flowers102:

[0229] Flowers102 is an image classification dataset containing 102 flower species. The number of images per category ranges from 40 to 258, totaling 8,189 images. This dataset is primarily used for flower classification and fine-grained recognition tasks. The diverse variety of flowers in the images makes it suitable for studying fine-grained image recognition tasks.

[0230] 2.5FGVCAircraft:

[0231] The FGVCAircraft dataset focuses on aircraft recognition and contains 100 different aircraft categories. The dataset contains over 10,000 images, covering a wide range of aircraft models. The FGVCAircraft dataset is suitable for studying tasks such as fine-grained image classification and object recognition.

[0232] 3. Scene Recognition Dataset:

[0233] 3.1SUN397:

[0234] SUN397 is a large-scale scene recognition dataset containing over 100,000 images across 397 categories. Each category represents a unique scene, such as a beach, a classroom, or a forest. This dataset is widely used for scene classification and scene understanding tasks. The challenges lie in the high visual similarity between categories and the complex and diverse backgrounds of the images included in the samples.

[0235] 4. Action Recognition Dataset:

[0236] 4.1UCF101:

[0237] UCF101 is a dataset of 101 different types of action videos, covering categories such as daily activities, sports, and musical instrument performances. With over 13,000 videos, it is used for tasks such as action recognition and behavior identification, making it a very popular benchmark dataset in the field of action classification.

[0238] 5. Texture Dataset:

[0239] 5.1DTD (Describable Textures Dataset):

[0240] DTD is a texture dataset containing 5,640 images across 47 categories. Each category is grouped according to visual descriptions, such as "striped" and "net-like." DTD is widely used to study texture classification, image retrieval, and other problems, and is suitable for recognizing subtle differences.

[0241] 6. Satellite Imagery Datasets:

[0242] 6.1 EuroSAT:

[0243] EuroSAT is a satellite image classification dataset containing more than 27,000 images captured by the Sentinel-2 satellite, covering 10 different geographic land cover classes, such as residential areas, forests, and farmland. This dataset is suitable for remote sensing image classification tasks and is used to evaluate the model's ability to identify different land cover types.

[0244] 7. Domain Generalization Dataset:

[0245] For domain generalization, the following ImageNet variants are used as target datasets:

[0246] 7.1ImageNet-V2: An extension of the original ImageNet dataset, designed to test the model's generalization ability on unseen new data.

[0247] 7.2ImageNet-Sketch: Contains hand-drawn sketch images of the same categories as ImageNet, used to test the recognition performance of the model on non-real-world images.

[0248] 7.3ImageNet-A: This is a specially collected dataset of "difficult-to-classify" images. Models trained on ImageNet perform poorly on these images and are often used to evaluate the robustness of models.

[0249] 7.4ImageNet-R: Contains some stylized images, such as paintings and cartoons, and is suitable for evaluating the model's generalization ability on non-standard image styles.

[0250] Through these datasets, we can comprehensively evaluate the performance of the model on different tasks and different data distributions, especially in scenarios such as generalization from basic categories to new categories, cross-dataset evaluation, and domain generalization.

[0251] c. Experimental implementation details:

[0252] The present invention uses the pre-trained ViT-B / 16CLIP model as the backbone network. By default, the length of global visual cues, fine-grained visual cues, and class-specific text cues are all set to 4, and the depth is set to 12 by default. The suppression factor ((λ)) is set to 0.1 by default and can be adjusted based on the validation set. For all tasks, the batch size is 4, the learning rate is 0.003, and SGD is used as the optimizer for model training. For each downstream task, the number of training rounds is optimized, and ImageNet is trained for 3 rounds with a learning rate of 0.002. The experimental results are the average of 3 random seeds. All experiments were performed on a single NVIDIA RTX 3090 GPU.

[0253] 2. Generalization experiment from basic categories to new categories:

[0254] We compare FGMPL with five baseline models: zero-shot CLIP, CoOp, CoCoOp, MaPLe, and ALIGN. Table 1 shows the generalization results from base classes to novel classes after 16 epochs of training on 11 datasets.

[0255] Table 1:

[0256]

[0257]

[0258]

[0259] As shown in Table 1, compared with the baseline of basic to new generalization, FGMPL learns dual-granularity visual cues and class-aware cues, showing stronger generalization results than existing methods on 11 recognition datasets. The best results are shown in bold, and the second best results are underlined. H refers to the harmonic mean. According to Table 1,

[0260] a. Performance evaluation on basic categories:

[0261] As shown in Table 1, our method significantly outperforms MaPLe without sacrificing generalization ability, achieving a 2.19% average accuracy improvement. It also achieves state-of-the-art performance on the baseline categories across all datasets, surpassing the state-of-the-art ALIGN by 1.09%. Unlike the learnable vectors in the multimodal cues proposed by MaPLe and ALIGN, which influence the classifier of the vision-language model, our proposed multimodal cue learning framework directly guides the model to capture more discriminative features, thereby obtaining more adaptable discriminative features for downstream image recognition tasks, achieving higher accuracy.

[0262] b. Generalization to unseen categories:

[0263] Across all 11 datasets, our FGMPL method achieved the best performance on both new categories and H-scores, with average accuracy improvements of 2.54% and 1.88%, respectively. We attribute this success to the dual-granularity multimodal cue tuning strategy, where the dual-granularity multimodal cue enhances the ability to recognize diverse visual concepts and plays a key role in unseen category prediction. In addition, for datasets with small intra-class variance (such as Flowers102 and FGVCAircraft), FGMPL significantly outperforms MaPLe and ALIGN. We believe that this improvement is due to the fine-grained visual cues in FGMPL, which can capture more subtle differences, thereby improving the efficiency of fine-grained image classification.

[0264] 3. Cross-dataset evaluation experiment:

[0265] Cross-dataset evaluation is more challenging than generalization from the base to novel categories. FGMPL was trained on 1000 categories from ImageNet and directly transferred to 10 other datasets. The comparative results are shown in Table 2. Overall, FGMPL achieved the best performance on both the source and target datasets, achieving an average accuracy of 68.21%, outperforming MaPLe and ALIGN by 1.91% and 1.06%, respectively. This demonstrates that the dual-granularity visual cues and class-specific textual cues in FGMPL promote better generalization.

[0266] Table 2:

[0267]

[0268] 4. Domain Generalization Experiment

[0269] Table 3 shows the classification accuracy on ImageNet and the target domain dataset. In addition to the previously mentioned methods, this method is also compared with VPT and UPT. It can be seen that the FGMPL method performs well and significantly improves the adaptability of the visual language model on different downstream image recognition datasets.

[0270] Table 3:

[0271]

[0272]

[0273] 5. Ablation Experiment

[0274] To verify the effectiveness of each module in the proposed method, we conducted ablation experiments on the ImageNet dataset using 16 training samples per class, removing key components such as the dual-granularity visual cue module and the multimodal cue generation module. The following is an analysis of the experimental results for each module:

[0275] a. Sensitivity analysis of inhibitors:

[0276] Table 4 shows the accuracy results under different suppression factor (λ) settings. The results show that keeping (λ) small (such as 0.1) can achieve better performance than not using the suppression module (λ = 1). The specific results are shown in Table 4.

[0277] Table 4:

[0278] λ 0.05 0.1 0.3 0.7 1.0 Accuracy 72.34 73.61 71.32 70.48 70.45

[0279] b. Effectiveness of different components:

[0280] FGMPL primarily consists of the following components: Global Visual Cues (GVP), Fine-Grained Visual Cues (FG-VP), Class-Specific Textual Cues (CATP), and Multimodal Cues Generation and Interaction (MMPG). Ablation experiments were conducted on each component, and the results are shown in Table 5. The introduction of GVP improved performance by 0.37%, and the use of FG-VP to enhance fine-grained difference information improved performance by 1.05%. These results demonstrate that combining CATP with GVP and FG-VP can yield a more comprehensive class-specific visual difference description, complementing image features and thus improving performance. Furthermore, the addition of MMPG further improves performance, demonstrating that generating multimodal cues from a shared semantic space helps promote cross-modal interaction and alignment.

[0281] Table 5:

[0282] Method Accuracy CLIP+Text Prompt 71.51 CLIP+CATP+GVP 71.88 CLIP+CATP+GVP+FG-VP 72.56 CLIP+CATP+GVP+FG-VP+MMPG 73.61

[0283] c. Validity of various constraints:

[0284] In order to evaluate the performance of the proposed multimodal cue tuning evaluation method, a constraint combination experiment was designed. The performance comparison results using different constraint combinations are shown in Table 6. Table 6 shows the performance comparison results using the class-specific global difference prediction constraint, i.e. After capturing global class-specific differences, the performance is improved by 1.07%; introducing fine-grained region prediction constraints, i.e. After that, the recognition performance was significantly improved by 3.45%.

[0285] Table 6:

[0286]

[0287] 6. Parameter Experiment:

[0288] The parameter experiments of α, β and γ were carried out, and the results are shown as follows: Figure 3 As shown in Figure 2, experiments show that FGMPL's performance is somewhat sensitive to parameter changes. The weight parameters of each component are relatively close, reflecting the effectiveness of the multimodal cue learning framework. In the experiments, the default values ​​of α, β, and γ were set to 0.5, 0.4, and 0.5, respectively.

[0289] The above experimental results demonstrate the effectiveness and superiority of the FGMPL method in generalization from basic categories to new categories, cross-dataset evaluation, domain generalization, and different module components.

[0290] The present invention significantly improves the ability of visual cues to capture global discriminative information and fine-grained class-specific features through the design of dual-granularity visual cues. Simultaneously, the semantic alignment of visual and textual cues is effectively enhanced through the visual-textual cue tuning module. Ultimately, through joint optimization, visual and textual cues can fully interact in a shared semantic space, improving the model's performance in multimodal tasks. Furthermore, while ensuring efficient training, the present invention freezes the encoder of the pre-trained model and trains only the cue learning module, significantly reducing the number of training parameters and improving the model's performance in downstream tasks.

[0291] The present invention specifically proposes a fine-grained multimodal cue learning framework, which enables the pre-trained visual language model (such as CLIP) to more sensitively capture class-specific differences, especially fine-grained distinguishing details, when processing downstream tasks through cue learning. The framework guides the visual language model to focus on the salient features and subtle difference features of the target object. With a small amount of training samples, adaptive discriminative features are obtained, thereby improving model performance. The present invention designs a visual-text cue tuning evaluator to ensure cross-modal interaction and collaborative reasoning between visual cues and text cues, thereby fully utilizing the cross-modal information representation capabilities of the CLIP model. The core idea of ​​the framework is to explicitly guide the model to focus on class-specific differences in the input through dual-granularity visual cues and class-specific text cues, and to improve the collaborative reasoning capabilities between visual and text cues through cross-modal interaction. Specifically, visual cues obtain class-specific differences by parsing the semantic features inferred by the image encoder, while text cues convert class names into class-specific descriptions to help the model understand visual concepts. The multimodal cue generation module generates visual and textual cues in a unified semantic space. Furthermore, a multimodal cue fine-tuning evaluation method is proposed to promote alignment and collaborative reasoning between visual and textual cues. In this way, the fine-grained multimodal cue learning framework can effectively improve the adaptability of pre-trained models to downstream tasks.

[0292] Furthermore, experimental verification was conducted on 11 widely used datasets. The experimental results show that this method has obvious advantages over other hint tuning methods in multiple downstream tasks. The specific beneficial effects are as follows:

[0293] 1) Improved classification accuracy: By combining dual-granularity visual cues with class-specific textual cues, the model's generalization ability to different datasets is significantly improved.

[0294] 2) Cross-modal alignment: Through the multimodal cue generation module, the interaction of visual and textual cues in a shared semantic space further improves the classification performance of complex images.

[0295] 3) Improved model efficiency: By freezing the encoder part of the pre-trained model and optimizing only the prompt-related parameters, training overhead is reduced and inference speed is improved.

[0296] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the scope of protection of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0297] A system for implementing any of the above-mentioned fine-grained multimodal prompt learning methods based on a visual language pre-training model in an embodiment of the present invention includes: a dual-granularity visual prompt module, a text prompt module, a visual-text prompt alignment generation module, and a visual-text prompt tuning module;

[0298] The dual-granularity visual cue module is used to provide: global visual cues for global high-level semantic information and fine-grained visual cues for extracting subtle differences between categories. The global visual cues are used to extract global semantic information of the input image, and the fine-grained visual cues are used to extract subtle differences between visual categories.

[0299] The text prompt module is used to: generate text prompts that are aware of category labels;

[0300] The visual-textual prompt alignment generation module is used to: perform cross-modal alignment and collaborative reasoning of visual and textual prompts to generate consistent semantic representations. The visual prompts include global visual prompts and fine-grained visual prompts.

[0301] The visual-textual cue tuning module is used to optimize the performance of visual cues and textual cues in downstream tasks based on loss functions of different dimensions.

[0302] In another embodiment, the following modules are included:

[0303] 1) Text Encoder: It is used to encode the input text. The text is generated by combining the context vector and the class label features. The parameters in the text encoder are frozen ( Figure 2 ), that is, the model does not update these parameters in downstream tasks.

[0304] 2) Visual Encoder: This encodes the input image and extracts both global and fine-grained features. The visual encoder consists of two branches: ① Global Visual Cue Branch: This is responsible for extracting global visual features from the input image. ② Fine-grained Visual Cue Branch: This is responsible for capturing subtle category differences in the image and capturing detailed regions through a suppression mechanism.

[0305] 3) Transformer layer: used to further refine and adjust the cues from vision and text. Figure 2 The flame symbol in indicates that the parameters of this layer are trainable.

[0306] 4) Multimodal prompt generation module: used to map visual and textual prompts into a shared semantic space to ensure cross-modal information interaction.

[0307] 5) Hint embedding module: This module integrates the generated hint embeddings and ultimately uses them for downstream image classification tasks.

[0308] 6) Multimodal Interaction Module and Decoding Module: This module includes two prompt branches and a multimodal prompt generation unit. Specifically:

[0309] ① Global visual cue branch: extracts global visual features of the input image.

[0310] ② Fine-grained visual cue branch: forces the model to capture fine-grained local information by randomly suppressing parts of the image area.

[0311] ③ Class-specific text hint branch: extracts class labels and context hints through a text encoder.

[0312] 7) Decoding Module: After processing through the Transformer layer and the multimodal cue generation module, the decoder outputs the category prediction. The decoding process uses the cue embedding obtained from the visual and textual cues.

[0313] 8) Alignment of global visual cues with class-specific textual cues: This module is responsible for establishing cross-modal interactions between visual and textual cues to improve classification accuracy through a shared semantic space.

[0314] It should be noted that the beneficial effects of the system for implementing any of the above-mentioned methods for learning a fine-grained multimodal prompt based on a visual language pre-training model provided by the above-mentioned embodiment are the same as the beneficial effects of the above-mentioned method for learning a fine-grained multimodal prompt based on a visual language pre-training model, which will not be repeated here. In addition, when the system provided by the above-mentioned embodiment realizes its functions, it only uses the division of the above-mentioned functional modules as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided by the above-mentioned embodiment belong to the same concept, and their specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0315] Among them, the system of the present invention for implementing any of the above-mentioned fine-grained multimodal prompt learning methods based on a visual language pre-training model can be a computer program (including program code) running on a computer device. For example, the system of the present invention for implementing any of the above-mentioned fine-grained multimodal prompt learning methods based on a visual language pre-training model is an application software that can be used to execute the corresponding steps in the fine-grained multimodal prompt learning method based on a visual language pre-training model of the present invention.

[0316] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any one of the above-mentioned methods for learning fine-grained multimodal prompts based on a visual language pre-training model is implemented. In other words, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; a memory for storing a computer program; and a processor for executing, by calling a computer program, a method for learning fine-grained multimodal prompts based on a visual language pre-training model as shown in any embodiment of the present invention. The electronic device may also be a terminal device, which may be any device that can install applications, including at least one of a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, and a smart car device.

[0317] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, which, when executed by a processor, implements any of the above-mentioned fine-grained multimodal prompt learning methods based on a visual language pre-training model.

[0318] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0319] In an exemplary embodiment, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform any of the aforementioned methods for learning fine-grained multimodal prompts based on a visual-language pre-trained model.

[0320] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0321] The above description is merely a preferred embodiment of the present invention and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present invention is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the present invention.

[0322] Those skilled in the art will appreciate that the present invention may be implemented as a system, method, or computer program product. Therefore, the present invention may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the present invention may be implemented in the form of a computer program product embodied in one or more computer-readable media containing computer-readable program code.

[0323] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A fine-grained multimodal cue learning method based on a visual language pre-training model, characterized by: include: S1. Construct a multimodal cue learning vector for learning visual and text features: S101. Construct a fine-grained multimodal cue learning model based on a visual language pre-training model. The fine-grained multimodal cue learning model includes visual cues and text cues. The fine-grained multimodal cue learning model is used to learn visual cues and text cues. Specifically, the image encoder and text encoder of the visual language pre-training model are used to process the input image and K category labels to generate a learnable visual cue V. i and learnable text prompts T i , where the input image is denoted as I∈R W×H×C , K is the total number of category labels, H is the height of the input image, W is the width of the input image, C is the channel of the input image, and i is the network layer index; S102, the learnable visual cue V i Designing global visual cues for global high-level semantic information and fine-grained visual cues for extracting subtle differences between categories. The global visual cues are used to extract global semantic information of the input image, and the fine-grained visual cues are used to extract subtle differences between visual categories. S103, the learnable text prompt T i Using category label-aware cue learning, the category label is represented by an embedded vector, and the category label is used as an input of the learnable text cue to generate category-differentiated text features; S104, generating the global visual cue, the fine-grained visual cue, and the learnable text cue in a shared latent semantic space, so that the fine-grained multimodal cue learning model can achieve cross-modal information interaction and alignment, wherein the multimodal cue learning vector includes the global visual cue, the fine-grained visual cue, and the learnable text cue; S2. Generate visual features and category-differentiated text features based on the fine-grained multimodal cue learning model: S201, generating category-specific text features by the text encoder: In formula (1), w represents the embedding vector of the category label, T b Represents text prompts, category-differentiated text features Z k Generated by the text encoder L(·); S202: Generate global visual features and fine-grained features of the input image through a visual encoder: In formula (2), e represents the image block embedding, V b Represents visual cues, features X include fusion features of global visual features and fine-grained features, and features X are generated by the image encoder I(·); S3. Optimize the multimodal cue learning vector based on the cosine similarity loss function: The learnable visual cue and the learnable text cue are optimized based on the cosine similarity loss function, and the similarity difference between the visual feature and the category-differentiated text feature is calculated by minimizing the similarity difference between the visual feature and the category-differentiated text feature, and the probability of the k-th category label is calculated by formula (3) Among them, τ is a fixed hyperparameter, sim(x,Z k ) is the similarity between the visual feature and the category-differentiating text feature.

2. A fine-grained multimodal cue learning method based on a visual language pre-training model according to claim 1, characterized in that: In S102, the process of obtaining the global visual features includes: The global visual cue branch extracts the global visual features of the input image through formula (4): [G i ,C i ,E i ]=I i ([G i-1 ,C i-1 ,E i-1 ]) (4) Where i is the layer index, I i represents the i-th visual encoder layer, C i and E i Represent the [CLASS] embedding and image patch embedding of each image respectively.

3. The fine-grained multimodal cue learning method based on a visual language pre-training model according to claim 2 is characterized in that: In S102, the process of obtaining fine-grained visual cues includes: The fine-grained visual cue branch sets the activation feature map A of the input image c ∈R H×W , where H and W represent the height and width of the input image respectively, the image block size is P, and the number of image blocks is: The set P of all image patches in the activation feature map c for: in, Represents the block at row m and column n on the activation feature map; Based on the set mask matrix M c , perform random mask operations on the image blocks in the activation feature map, M c ∈R H×W , and set the mask matrix M c for: Among them, Bernoulli distribution represents the probability distribution of mask operation, P patch represents the probability of masking the image block, M c The mask of the highest activation value position in (i, j) is set to 0, which is specifically expressed by formula (7): M c (i,j)=0,ifA c (i,j)=max(A c ) (7) Through the activated feature map A′ after suppression c Generate fine-grained visual cues F, F = {F j ∈R d ∣∣j∈N,1≤j≤n f }, specifically: Where λ represents the inhibition factor; Fine-grained visual cues are input to the fine-grained vision branch via formula (9): [F i ,C i ,E′ i ]=I i ([F i-1 ,C i-1 ,E′ i-1 ]) (9) Among them, E′ i Represents the suppressed image patch embedding.

4. The fine-grained multimodal cue learning method based on a visual language pre-training model according to claim 3 is characterized in that: In S103, the class-aware text prompt is designed to adapt to downstream tasks by automatically generating text prompts that semantically match the visual class-specific concepts. This is specifically accomplished through formula (10): [_,W i ]=L i ([T i-1 ,W i-1 ]) (10) Among them, the text prompt of the k-th category label is T k ={u1,…,u M ,c i },{u1,…,u M } represents the learnable text context vector, and the word embedding vector of the text class tag is c i , L i is the i-th layer of the text encoder, W i is the word embedding representation.

5. The fine-grained multimodal cue learning method based on a visual language pre-training model according to claim 4 is characterized in that: In S104, generating the global visual cue, the fine-grained visual cue, and the learnable text cue in a shared latent semantic space further includes: A multimodal cue generation module is proposed to map global and fine-grained visual cues in a shared semantic space to each layer of the visual encoder through a linear layer, which is specifically accomplished by formula (11): Where i is the layer index, N is the total number of layers, and the length of the text prompt, global visual prompt and fine-grained visual prompt of each layer is the same. A linear projection layer representing global visual cues, Linear projection layer for representing fine-grained visual cues.

6. The fine-grained multimodal cue learning method based on a visual language pre-training model according to claim 5 is characterized in that: In S3, the calculation is performed by minimizing the similarity difference between the visual features and the category-differentiated text features, including: Design a multimodal cue tuning evaluation method, using visual embedding and class-aware text embedding of global and fine-grained visual cue branches, setting y to be the true label, and optimizing the visual classification loss Enables fine-grained multimodal cue learning models to find subtle differences between related categories and optimize visual classification losses for: Assume that the category-specific visual embedding and text embedding are E G and E T , using formula (13) to calculate the category-specific global difference prediction, namely the global difference loss To align the visual embeddings with the category-specific text embeddings: Among them, E G represents the global visual embedding, represents the text embedding corresponding to category i, and τ is a hyperparameter that controls the sharpness of the distribution; The visual embedding and text embedding pair of the fine-grained visual cue branch is represented as (E F ,E T ), calculate the fine-grained difference loss The calculation formula is: in, represents the text embedding of the corresponding category i, τ is a hyperparameter that controls the sharpness of the distribution, and is supervised by To enhance the ability to extract subtle information features; Will and For optimization, the overall loss function formula is: Among them, α is The weight hyperparameter, β is The weight hyperparameter, γ is The weight hyperparameters.

7. A system for implementing the fine-grained multimodal cue learning method based on a visual language pre-training model as described in any one of claims 1 to 6, characterized in that: The system includes: a dual-granularity visual prompt module, a text prompt module, a visual text prompt alignment generation module and a visual text prompt tuning module; The dual-granularity visual cue module is used to provide: global visual cues for global high-level semantic information and fine-grained visual cues for extracting subtle differences between categories. The global visual cues are used to extract global semantic information of the input image, and the fine-grained visual cues are used to extract subtle differences between visual categories. The text prompt module is used to: generate text prompts for category label perception; The visual-textual prompt alignment generation module is used to: perform cross-modal alignment and collaborative reasoning on visual prompts and text prompts to generate consistent semantic representations, where the visual prompts include the global visual prompts and the fine-grained visual prompts; The visual-textual prompt tuning module is used to optimize the performance of visual prompts and textual prompts in downstream tasks based on loss functions of different dimensions.

8. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for learning fine-grained multimodal prompts based on a visual language pre-training model according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the fine-grained multimodal prompt learning method based on a visual language pre-training model as described in any one of claims 1 to 6.

Images