Image classification method of visual language model based on attribute anchor prompt word fine tuning

Abstract

This invention provides a visual language model image classification method based on attribute anchor cue word fine-tuning, relating to the fields of computer vision, deep learning, and visual language model technology. The invention generates attribute text based on the image dataset to be identified and constructs a training sample set; it constructs hybrid cue words, inputs the hybrid cue words and the image dataset to be identified into a visual language model, and identifies the category to which each image in the image dataset belongs. Under the condition of freezing the pre-trained visual language model, this invention utilizes an efficient parameter fine-tuning method for category attribute-enhanced text cueing to solve the problems of unclear semantics and weak generalization ability in existing cue learning, ensuring that attribute information is not overwhelmed by category names and improving image recognition performance on both base and new classes in multi-dataset scenarios.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, deep learning, and visual language model technology, and particularly relates to a visual language model image classification method based on attribute anchor cue word fine-tuning. Background Technology

[0002] With the development of large-scale image-text datasets and contrastive learning techniques, pre-trained visual language models have demonstrated strong transfer capabilities in zero-shot and few-shot image classification tasks. A typical approach is to freeze the pre-trained image encoder and text encoder, fill the category name into a fixed template sentence "a photo of a {CLASS}" from the text side prompts, obtain the corresponding category text features, and then calculate the similarity with the image features in the same feature space to complete the classification.

[0003] To further improve the model's performance on downstream tasks, a common approach is to fine-tune the pre-trained model. However, this method has significant limitations: firstly, it requires a large number of labeled samples, making it difficult to adapt to scenarios with few or even zero samples; secondly, updating some parameters of a large-scale pre-trained model incurs high training and deployment costs and is prone to overfitting, weakening its generalization ability across datasets. Another approach focuses on improving the text prompting level. These methods typically freeze the pre-trained image and text encoders and make lightweight adjustments to the text input by inserting a small number of learnable soft-cue words before the category names, thus realigning image and text features within the original feature space. However, existing soft-cue learning methods lack explicit semantics, leading to the following problems: firstly, limited generalization ability to new categories or out-of-distribution data; and secondly, the internal structure of the prompts is uninterpretable. Some studies have attempted to introduce richer semantic information into the prompts, such as using category attribute descriptions and contextual sentences to enhance the text input, enabling the model to refer to more semantic features of the category when making decisions. These methods improve the expressiveness of prompts to some extent, but they usually involve directly splicing long sentences into the prompts, lacking a mechanism for uniformly compressing and representing multiple sentences.

[0004] In summary, existing technologies improve image recognition performance by fine-tuning prompt words while freezing the backbone of the pre-trained visual language model. However, problems remain, such as template design relying on experience, unclear prompt semantics, and insufficient generalization ability for new classes and complex scenes. Constructing a prompt representation with a clear semantic structure that is easy to generalize and reuse across datasets while freezing the backbone parameters of the pre-trained model remains an unsolved technical challenge. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this paper proposes to build stable soft cues using attribute text while freezing the backbone of the pre-trained visual language model. This ensures that attribute information is not overwhelmed by category names and improves image recognition performance in multi-dataset scenarios.

[0006] To address the aforementioned technical problems, in a first aspect, the present invention provides a visual language model image classification method based on attribute anchor cue word fine-tuning, comprising the following steps:

[0007] Obtain the image dataset to be identified, generate attribute text based on the image dataset, and construct a training sample set based on the attribute text;

[0008] Construct hybrid prompt words, the hybrid prompt words including soft prefixes Category name embedding and aggregation embedding ;

[0009] The mixed prompts and the image dataset to be identified are input into the visual language model to identify the category of each image in the image dataset;

[0010] The specific method for constructing mixed prompt words is as follows:

[0011] Initialize learnable soft prefixes ;

[0012] Construct an attribute compression module and train it using the training sample set to obtain the trained attribute compression module:

[0013] A static attribute pool is constructed based on the trained attribute compression module. The static attribute pool includes the soft attribute token tensor corresponding to each category in the image dataset. ;

[0014] Obtain the category name token corresponding to each category name in the training sample set and input it into the text encoder to obtain the category name embedding for each category. ;

[0015] Construct a cross-attention aggregation module to aggregate soft attribute token tensors from the static attribute pool. and category name embedding Aggregation, resulting in aggregated embeddings ;

[0016] Learnable soft prefixes Aggregation and Embedding and category name embedding By combining the keywords, we obtain a mixed prompt.

[0017] For cross-attention aggregation modules and learnable soft prefixes Fine-tune the parameters until the preset number of iterations or the loss function converges, and obtain the trained cross-attention module and the optimized soft prefix.

[0018] Furthermore, the specific method for generating attribute text based on image datasets is as follows:

[0019] Based on the image dataset, construct cue words, input the cue words into a large language model, and use the large language model to generate attribute text sets for each category in the image dataset. Each category's attribute text set includes multiple attribute texts, which are connected by concatenation symbols.

[0020] The specific method for constructing a training sample set based on attribute text is as follows:

[0021] Summarize the attribute text sets of all categories, and remove duplicate attribute text to obtain the following: The training sample set of attribute text.

[0022] Furthermore, the attribute compression module includes a first normalization layer, a first fully connected layer, a GELU activation layer, and a second fully connected layer;

[0023] The specific method for training the attribute compression module using the training sample set is as follows:

[0024] The batches will include The training sample set of attribute texts is input in parallel into the text encoder of the pre-trained visual language model, and the parameters of the text encoder are frozen. Each batch of attribute texts is encoded into a token embedding sequence by a text segmenter and fed into the text encoder to extract the text features of the attribute texts. The text features of the attribute texts are used as the teacher text features.

[0025] The token embedding sequence is compressed into a soft-attribute token embedding sequence of length K based on the attribute compression module, thus obtaining a soft-attribute token tensor;

[0026] The soft attribute token tensor is input into the text encoder to obtain the feature vector after attribute compression. The feature vector after attribute compression is then processed... Norm normalization operation yields student text features;

[0027] A bidirectional main loss function is constructed based on teacher and student text features, and the attribute compression module is trained to obtain the trained attribute compression module. The bidirectional main loss function includes the total loss function from student to teacher and the total loss function from teacher to student.

[0028] Furthermore, the specific method for constructing a static attribute pool based on the trained attribute compression module is as follows:

[0029] Load the trained attribute compression module and text encoder; freeze the parameters of the trained attribute compression module and text encoder completely without performing gradient updates;

[0030] Traverse all categories in the image dataset, input the attribute text set corresponding to each category into the frozen attribute compression module in parallel, and calculate the soft attribute token tensor corresponding to each category.

[0031] Each category's corresponding soft attribute token tensor is stored using the category as the index, constructing a category-level static attribute pool, where the category... The soft property token tensor is .

[0032] Furthermore, the cross-attention aggregation module includes a cross-attention mechanism, layer normalization, and a feedforward neural network;

[0033] The cross-attention aggregation module is used to aggregate the soft attribute token tensors in the static attribute pool. and category name embedding Aggregation, resulting in aggregated embeddings The specific method is as follows:

[0034] Initialize a set of random, learnable vectors as query vectors. ;

[0035] For category Retrieve the corresponding soft attribute token tensor from the static attribute pool. , the soft attribute token tensor and category name embedding Concatenate the sequences to build a key-value pool. ;

[0036] Utilizing a multi-head attention mechanism, based on the query vector and key-value pool The attention output is calculated and processed through layer normalization and a feedforward neural network to obtain the aggregated embedding. .

[0037] Furthermore, regarding the cross-attention aggregation module and learnable soft prefixes... The specific method for fine-tuning parameters is as follows:

[0038] Extracting text feature vectors based on mixed prompt words ;

[0039] For each category in the image dataset, 80 standard handwritten templates provided by ImageNet are obtained. The category name of each category is filled into the standard handwritten template, and then input into the text encoder to extract features. Each category yields 80 standard handwritten template feature vectors.

[0040] Calculate the mean of the feature vectors of 80 handwritten standard templates for each category, and then perform... Normalization yields the semantic anchor feature vector for each category. ;

[0041] Using semantic anchor feature vectors Text feature vectors Constraints are imposed to prevent the learnable parameters from becoming too divergent. Finally, the parameters of the cross-attention aggregation module and the soft cue prefix are fine-tuned by comparing the loss function.

[0042] During training, only the parameters of the cross-attention aggregation module and the soft prefix are updated until the training reaches the preset number of iterations or the loss function converges, thus obtaining the final trained parameters of the cross-attention aggregation module and the optimized soft prefix.

[0043] During the inference phase, the trained cross-attention aggregation module is used to generate aggregated embeddings. The mixed prompt words are then concatenated with category name embeddings and optimized soft prefixes to generate hybrid prompt words. The hybrid prompt words and the image dataset to be identified are then input into a pre-trained visual language model to identify the category to which each image in the image dataset belongs.

[0044] Furthermore, the total loss function includes contrastive loss and semantic regularization loss; the contrastive loss includes image-to-text loss and text-to-image loss; the strength of the semantic regularization loss is adjusted by regularization weights;

[0045] The regularization weights employ a linear warm-up strategy, increasing linearly from 0 to a preset value in the first few training rounds and then remaining unchanged thereafter.

[0046] Secondly, this application proposes an electronic device, including: one or more processors, and a memory for storing instructions, which, when executed by the one or more processors, cause the one or more processors to perform the visual language model image classification method based on attribute anchor cue word fine-tuning.

[0047] Thirdly, this application proposes a computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform the visual language model image classification method based on attribute anchor cue word fine-tuning.

[0048] Fourthly, this application proposes a computer program product, including a computer program or instructions that, when executed by a processor, implement the aforementioned visual language model image classification method based on attribute anchor cue word fine-tuning.

[0049] The beneficial effects of adopting the above technical solution are as follows: This invention provides a visual language model image classification method based on attribute anchor cue word fine-tuning. It constructs an attribute compression module and a cross-attention aggregation module. The attribute compression module performs lightweight compression of attribute text; the cross-attention aggregation module ensures that the model can extract the most critical semantic information from long attribute descriptions. Under the condition of freezing the pre-trained visual language model, this invention uses an efficient parameter fine-tuning method for category attribute-enhanced text cues to solve the problems of unclear semantics and weak generalization ability in existing cue learning. This ensures that attribute information is not overwhelmed by category names and improves image classification performance on both base and new classes in multi-dataset scenarios. Attached Figure Description

[0050] Figure 1 The overall architecture diagram of the complete attribute anchor point prompt word fine-tuning method provided in Embodiment 1 of the present invention;

[0051] Figure 2 The present invention provides a visual comparison of GradCAM based on different hinting fine-tuning methods on the Caltech101 dataset, according to Embodiment 1 of this invention. Detailed Implementation

[0052] The specific implementation methods of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0053] Example 1:

[0054] A visual language model image classification method based on attribute anchor cue word fine-tuning, such as... Figure 1 As shown, it includes the following steps:

[0055] Step 1: Obtain the image dataset to be identified, generate attribute text based on the image dataset, and construct a training sample set based on the attribute text;

[0056] Step 1.1: Obtain the image dataset to be identified, generate attribute text from the image dataset, and obtain the attribute text set for each category;

[0057] A large language model is used to generate a set of attribute texts for each category in the image dataset. Each set of attribute texts for each category includes multiple attribute texts connected by concatenation symbols.

[0058] This embodiment selects seven representative publicly available image recognition benchmark datasets, including the Caltech101 dataset, OxfordPets dataset, OxfordFlowers dataset, Food-101 dataset, FGVCAircraft dataset, DTD dataset, and EuroSAT dataset, covering various visual tasks such as general object recognition, fine-grained classification, texture and scene recognition. The prompt is: For the category {CLASS} within the {TYPE} type category, please generate five highly distinguishable natural language visual attributes. Each attribute text must describe the clearly observable visual features of that category in the image, where {TYPE} is the type category and {CLASS} is the category name. The {TYPE} type category is the parent category of the {CLASS} category name; for example, the parent category of the category name "apple" could be "fruit" or "mobile device." By incorporating type categories into the prompt, the large language model can more accurately output attribute text.

[0059] In this embodiment, the prompt words are input into the GPT5.1 large language model, and attribute text is generated for each category in the 7 datasets. Each category yields 5 different attribute texts, which are connected by semicolons to obtain the attribute text set for each category.

[0060] Step 1.2: Summarize the attribute text sets of all categories, deduplicate the attribute text, and remove duplicate attribute text to obtain the following: Training sample set of attribute text;

[0061] Step 2: Construct the attribute compression module and train it using the training sample set to obtain the trained attribute compression module.

[0062] Step 2.1: Input the training sample set into the text encoder of the pre-trained visual language model in batches to construct teacher text features;

[0063] The batches will include The training sample set of attribute text is input in parallel into the text encoder of the frozen pre-trained visual language model to obtain the teacher's text features. The specific process is as follows:

[0064] First, a pre-trained visual language model's text encoder, such as CLIP's Transformer text tower, is loaded, and the parameters of the text encoder are completely frozen. Each batch of attribute text is encoded into a token embedding sequence by a tokenization text segmenter, and then fed into the frozen text encoder to extract the text features of the attribute text. The text features of the attribute text are used as teacher text features to constrain the soft token generation of the attribute compression module.

[0065] The text encoder used in this embodiment contains 12 layers, the first of which is a word embedding layer, used to output the token embedding sequence corresponding to all attribute text in each batch. Where T is the length of the token embedding sequence. D represents the number of attribute texts within a batch, and D represents the dimension of the token embedding sequence. The last layer of the text encoder is used to extract the text features of the attribute texts, and these text features are used as the teacher text features. .

[0066] Step 2.2: Construct an attribute compression module. Based on the attribute compression module, compress the token embedding sequence length to a soft attribute token embedding sequence length of length K to obtain a soft attribute token tensor.

[0067] The attribute compression module includes a first-layer normalization, a first fully connected layer, GELU activation, and a second fully connected layer. It inputs the token embedding sequences corresponding to all attribute texts within a batch into the attribute compression module in parallel, and processes the token embedding sequence for each attribute text within the batch. After being aggregated into sentence vectors by average pooling, the vectors then pass through a first normalization layer, a first fully connected layer, a GELU activation layer, and a second fully connected layer, mapping the dimension of the sentence vectors from D to [the desired dimension]. Then, after a reshape operation, it is transformed into K vectors of length D, and then normalized through a second layer to obtain K soft-attribute token tensors. The soft attribute token tensor of each attribute text in the batch By concatenating the components, we obtain the soft attribute token tensor for each batch of attribute text. .

[0068] Step 2.3: Construct student text features based on the soft attribute token tensor;

[0069] The soft attribute token tensor The input is fed into the frozen text encoder to obtain the feature vector processed by the attribute compression module. The feature vector processed by the attribute compression module is then... Norm normalization operation yields student text features. .

[0070] To ensure that student and teacher feature vectors are compared and learned on the same scale, this embodiment performs the following steps on the feature vectors after the attribute compression module: The norm normalization operation yields the student feature vector, which is used for subsequent loss calculation and comparative learning to train the attribute compression module.

[0071] Step 2.4: Construct a bidirectional main loss function based on teacher text features and student text features, and train the attribute compression module to obtain the trained attribute compression module;

[0072] A teacher-student model of knowledge distillation is employed to train the attribute compression module. To ensure that contrastive learning takes place at the same scale, teacher text features are compressed. conduct Norm normalization operation.

[0073] Constructing a bidirectional main loss function The attribute compression module is trained by comparing student text features with teacher text features, including the following steps:

[0074] Step 2.4.1: Construct the total loss function from student to teacher;

[0075] Construct a student-to-teacher direction similarity matrix. The first element in the student-to-teacher direction similarity matrix... Line number Column elements Indicates the first in the current training batch Student text features With the Teacher text features The normalized similarity between them is shown in the following formula:

[0076] ;

[0077] in, For temperature parameters;

[0078] For a batch of size M, the first... Student text features The first batch Teacher text features The first M-1 teacher text features are considered positive samples, and the remaining M-1 teacher text features are considered negative samples. The total loss function from student to teacher is then the average loss of all student text samples in the batch, as shown in the following formula:

[0079] ;

[0080] in, Indicates the first The target probability distribution vector corresponding to each student's text feature.

[0081] Step 2.4.2: Construct the total loss function from teacher to student;

[0082] Construct a teacher-to-student directional similarity matrix. The first element in the teacher-to-student directional similarity matrix... Line number Column elements Indicates the first in the current training batch Teacher text features With the Student text features The normalized similarity between them is shown in the following formula:

[0083] ;

[0084] The total loss from teacher to student is the average of the losses of all teacher text samples within the batch, as shown in the following formula:

[0085] ;

[0086] Step 2.4.3: Establish a bidirectional main loss function and introduce cosine constraint auxiliary loss to obtain the total loss function;

[0087] Bidirectional main loss function As shown in the formula below:

[0088] ;

[0089] Based on this, a cosine-constrained auxiliary loss is introduced. This is used to encourage student embeddings to maintain consistency in direction with the corresponding teacher embeddings, as shown in the following formula:

[0090] ;

[0091] Based on bidirectional main loss function Sum and cosine constraint auxiliary loss The total loss function is obtained. As shown in the formula below:

[0092] ;

[0093] in, This is the regularization coefficient, typically set to 0.1. During training, only the parameters of the attribute compression module (PCE) are updated, while all parameters of the text encoder are frozen.

[0094] Step 3: Construct a static attribute pool based on the trained attribute compression module to realize offline pre-computation of attribute features and reduce training overhead;

[0095] Step 3.1: Load the attribute compression module trained in Step 2 and the frozen text encoder of the visual language model; in all subsequent steps, completely freeze the parameters of the trained attribute compression module and the text encoder of the pre-trained visual language model, do not perform gradient updates, and use them only as feature extractors.

[0096] Step 3.2: Attribute feature pre-calculation;

[0097] Iterate through all categories in the image dataset, input the attribute text set corresponding to each category into the frozen attribute compression module in parallel, and calculate the soft attribute token tensor corresponding to each category.

[0098] Step 3.3: Static attribute pool storage;

[0099] Store the soft attribute token tensor corresponding to each category using the category as the index to build a category-level static attribute pool. Any category The soft property token tensor is ,in For category The number of attribute texts. The static attribute pool serves as a read-only key-value lookup source during subsequent training.

[0100] Step 4: Construct a cross-attention aggregation module to aggregate the soft attribute token tensor and category name information in the static attribute pool to obtain the attribute aggregation token embedding vector;

[0101] To fuse the soft attribute token tensor with class name information and convert it into a fixed-length cue summary, a cross-attention aggregation module with learnable parameters is constructed; at the same time, a learnable soft cue prefix vector is constructed as a prefix for the hybrid cue words.

[0102] Step 4.1: Initialize learnable soft prefixes ,in, This is the length of the soft prefix. Used to provide optimized contextual information at the very beginning of mixed prompt words;

[0103] Step 4.2: Construct a cross-attention aggregation module, including a cross-attention mechanism, layer normalization, and a feedforward neural network; use the cross-attention aggregation module to aggregate the soft attribute token tensor with the category name information to obtain the attribute aggregated token embedding vector. This includes the following steps:

[0104] Step 4.2.1: Initialize a set of random learnable vectors as query vectors. ,in, This is a hyperparameter representing the number of tokens after attribute aggregation;

[0105] Step 4.2.2: Based on the soft attribute token tensor and category name embedding Building a key-value pool:

[0106] For input category Obtain the corresponding soft attribute token tensor from the static attribute pool in step 3. At the same time, the categories The category name token is input into the frozen text encoder to obtain the category name embedding. . Soft attribute token tensor and category name embedding Concatenate the sequences to build a key-value pool. This design ensures that the aggregation process considers both attribute details and class name body names.

[0107] Step 4.2.3: Utilize a multi-head attention mechanism based on the query vector. and key-value pool The attention output is calculated, then processed through layer normalization and a feedforward neural network to obtain the aggregated embedding. ;

[0108] Step 5: Construct hybrid prompt words and extract text feature vectors based on the hybrid prompt words;

[0109] Step 5.1: For any class within the training batch Embed learnable soft cue prefixes Aggregation and Embedding and category name embedding By concatenating the text, we obtain mixed text prompts. ;

[0110] Step 5.2: Mix the prompt words Input a frozen text encoder to extract mixed prompt words based on the marker symbol [EOT]. The positional characteristics of the end, and perform Normalization yields categories Text feature vectors ;

[0111] Step 6: Construct semantic regularization constraints based on multiple template centers;

[0112] To prevent learnable soft prefixes from overfitting image features and losing textual semantics during fine-tuning, a regularization constraint based on multiple template centers is introduced.

[0113] Step 6.1: Calculate the center of the template;

[0114] For each category in the image dataset, the category name of each category was filled into the handwritten standard template using 80 standard templates provided by ImageNet, and then input into a frozen text encoder to extract features, resulting in 80 template feature vectors for each category.

[0115] Step 6.2: Constructing regularized anchor points;

[0116] Calculate the mean of the 80 template feature vectors for each category, and then perform... Normalization yields the semantic anchor feature vector for each category. , which represents the center of the text encoder’s inherent semantic representation for each category, and remains unchanged during training.

[0117] Step 7: Construct a total loss function that includes contrastive loss and regularization loss, and fine-tune the parameters of the cross-attention aggregation module and soft cue prefix;

[0118] Step 7.1: Construct contrast loss, including image-to-text loss and text-to-image loss;

[0119] For a training batch of size B, image features are extracted using a frozen image encoder. Calculate image features and the text feature vector of the current batch The similarity score is logits between the text and text. A symmetric cross-entropy loss function is used to calculate the classification loss for both image-to-text and text-to-image.

[0120] Step 7.1.1: Construct the image-to-text loss;

[0121] For the first in the batch The image, and its corresponding number One attribute text is considered a positive sample, and the remaining B-1 attribute texts are considered negative samples. The image-to-text loss is then expressed by the following formula:

[0122] ;

[0123] in, For temperature coefficient, Represented as the first Zhang negative sample, For the first Image features of the image For the first Text feature vector of an image, For the first Text feature vector of the image;

[0124] Step 7.1.2: Construct the text-to-image loss;

[0125] Similarly, for the first in a batch The attribute text, its corresponding first The images are considered as positive samples. The text-to-image loss is then expressed by the following formula:

[0126] ;

[0127] Step 7.1.3: Add the image-to-text loss and the text-to-image loss to obtain the contrast loss;

[0128] ;

[0129] Step 7.2: Construct the semantic regularization loss;

[0130] Calculate text features With the regularized semantic anchor feature vector generated in step 6 Cosine similarity between A regularization loss term is constructed to force text features to align with the original semantic centers in terms of orientation, as shown in the following formula:

[0131] ;

[0132] in, The preset edge threshold is used, for example, the edge threshold is set in this embodiment. The RELU function ensures that a penalty is only incurred when the similarity is below a threshold.

[0133] Step 7.3: Construct a joint loss function based on contrastive loss and semantic regularization loss. As shown in the formula below:

[0134] ;

[0135] in, For regularization weights;

[0136] To stabilize training, the weights are regularized. A linear warm-up strategy is adopted, that is, in the first few rounds of training, the weight values ​​are linearly increased from 0 to the preset hyperparameters. The parameters remain unchanged thereafter. During training, only the parameters of the cross-attention aggregation module and the soft prefix are updated until the training reaches the preset number of iterations or the loss function converges. At this point, the final trained cross-attention aggregation module and the optimized soft prefix can be obtained.

[0137] During the inference phase, the trained cross-attention aggregation module is used to generate aggregated embeddings. The mixed prompt words are then concatenated with category name embeddings and optimized soft prefixes to generate hybrid prompt words. The hybrid prompt words and the image dataset to be identified are then input into a pre-trained visual language model to identify the category to which each image in the image dataset belongs.

[0138] To verify the effectiveness and generalization ability of the visual language model image classification method based on attribute anchor cue word fine-tuning provided in this embodiment, experiments were conducted on seven publicly available image recognition benchmark datasets selected in this embodiment. The experiments were based on the PyTorch framework, with a single A100 GPU as the hardware environment. The backbone network used the pre-trained CLIP ViT-B / 16 model, and all its parameters were frozen.

[0139] This embodiment is based on the acquisition and lightweight compression of attribute text. First, a large language model (such as GPT-5) is used to generate 5 visually discriminative attribute phrases for each category. The prompt word templates for the 7 datasets are shown in Table 1.

[0140] Table 1. Prompt word templates for different datasets;

[0141]

[0142] The attribute compression module is trained, which consists of two fully connected layers. The input is the compressed mean vector of the entire sentence. The output is a compact sequence of soft-attribute tokens. As shown in the formula below:

[0143] ;

[0144] in, , These represent the weights of the two fully connected layers; in this embodiment, the number of output soft attribute token embeddings is set to K=2. Optimization is achieved through a bidirectional main loss function combined with cosine similarity loss. After training, the attribute sentences for each category are input into this module, calculated offline, and stored as a static attribute pool. , serving as a read-only key-value source for subsequent stages.

[0145] To adapt to downstream tasks, this embodiment initializes a dimension as follows: Learnable soft prefixes , 8. Construct a cross-attention aggregation module and define M learnable query vectors. For any category Soft property token tensor and category name embedding Concatenate the sequences to build a key-value pool. Utilizing multi-head attention mechanism Calculate the aggregate embedding This design ensures that the model can extract the most critical semantic information from long attribute descriptions.

[0146] This embodiment further describes the construction and training process of hybrid prompt words. During training, for each category... Dynamically constructing mixed input sequences This involves sequentially concatenating the soft prefix, aggregate attribute embedding vector, and class name embedding vector. This sequence is then directly input into a frozen text encoder, where the vector at the end [EOT] position is extracted and normalized to obtain the category. Text feature vectors To prevent semantic drift, this embodiment also introduces a semantic regularization mechanism: calculating the semantic regularization for each category using 80 standard handwritten templates provided by ImageNet. semantic anchors and constrain during training The text features do not deviate from the anchor point.

[0147] During the training of the cross-attention aggregation module, this embodiment introduces a learnable temperature parameter with an initial value of 0.05. The AdamW optimizer was used, and the regularization weights adopted a linear warm-up strategy, which increased linearly from 0 to the learning rate in the first 5 epochs, with a total of 20 training epochs.

[0148] This embodiment evaluates model performance during the inference phase, and provides a visual comparison of GradCAM using different hinting fine-tuning methods on the Caltech101 dataset, as shown below. Figure 2 As shown, using the trained attribute compression module and cross-attention aggregation module, the mixed prompt words are input into the text encoder to obtain text features, and the similarity between them and the image features of the test is calculated.

[0149] Following the academic standard partitioning protocol, all categories in the image dataset are divided into two mutually exclusive subsets in a 50%:50% ratio: a base class and a new class. The base class includes a base class training set, a base class validation set, and a base class test set. The base class validation set is used to validate the model's performance, and the base class test set is used to measure the model's performance improvement on known categories. All images in the new class are used as the new class test set to test the model's generalization ability on unseen categories.

[0150] During training, instead of using all images from the base class training set, 16 images are randomly selected from each category within the base class training set as a few-shot training set. In this embodiment, the image samples from each category in the base class are divided into a base class training set, a base class validation set, and a base class test set in a 50%:20%:30% ratio. During training, the model is periodically evaluated using the base class validation set to select the model parameters with the best performance. After completing the few-shot training, the model is directly transferred to the new class test set for testing.

[0151] The accuracy rates of the base class and the new class were calculated separately, and the harmonic mean (HM) was used as the core evaluation index. The experimental results are shown in Table 2. This method significantly improves the generalization recognition ability of the new class while maintaining the performance of the base class, and the HM of this method is significantly improved compared with the baseline.

[0152] Table 2 Base-to-New Generalization Experiment;

[0153]

[0154] Example 2:

[0155] This embodiment proposes an electronic device, including: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the visual language model image classification method based on attribute anchor cue word fine-tuning.

[0156] The electronic device may be a mobile phone, computer, or tablet computer, etc., and includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the visual language model image classification method based on attribute anchor cue word fine-tuning as described in the embodiments. It is understood that the electronic device may also include input / output (I / O) interfaces and communication components.

[0157] The processor is used to execute all or part of the steps in the visual language model image classification method based on attribute anchor cue word fine-tuning as described in the above embodiments. The memory is used to store various types of data, which may include, for example, instructions for any application or method in the electronic device, as well as application-related data.

[0158] The processor can be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic components, and is used to execute the visual language model image classification method based on attribute anchor cue word fine-tuning described in the above embodiments.

[0159] Example 3:

[0160] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0161] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the visual language model image classification method based on attribute anchor cue word fine-tuning described in various embodiments of this application.

[0162] The aforementioned storage media include: flash memory, hard disks, multimedia cards, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory), random access memory (RAM), static random-access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, disks, optical discs, servers, APP (Application) app stores, and other media capable of storing program verification codes. These media store computer programs, which, when executed by a processor, can implement the various steps of the aforementioned visual language model image classification method based on attribute anchor cue word fine-tuning.

[0163] Example 4:

[0164] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned visual language model image classification method based on attribute anchor cue word fine-tuning.

[0165] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0166] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0167] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of this disclosure and its equivalents, then the intent of this disclosure also includes these modifications and variations.

Claims

1. A visual language model image classification method based on attribute anchor cue word fine-tuning, characterized in that, Includes the following steps: Obtain the image dataset to be identified, generate attribute text based on the image dataset, and construct a training sample set based on the attribute text; Construct hybrid prompt words, including soft prefixes. Category name embedding and aggregation embedding ; The mixed prompts and the image dataset to be identified are input into the visual language model to identify the category of each image in the image dataset; The specific method for constructing mixed prompt words is as follows: Initialize learnable soft prefixes ; Construct an attribute compression module and train it using the training sample set to obtain the trained attribute compression module: A static attribute pool is constructed based on the trained attribute compression module. The static attribute pool includes the soft attribute token tensor corresponding to each category in the image dataset. ; Obtain the category name token corresponding to each category name in the training sample set and input it into the text encoder to obtain the category name embedding for each category. ; Construct a cross-attention aggregation module to aggregate the soft attribute token tensors in the static attribute pool. and category name embedding Aggregation, resulting in aggregated embeddings ; Learnable soft prefixes Aggregation and Embedding and category name embedding By combining the keywords, we obtain a mixed prompt. For cross-attention aggregation modules and learnable soft prefixes Fine-tune the parameters until the preset number of iterations or loss function convergence is reached, and obtain the trained cross-attention module and the optimized soft prefix.

2. The visual language model image classification method based on attribute anchor cue word fine-tuning according to claim 1, characterized in that, The specific method for generating attribute text based on image datasets is as follows: Based on the image dataset, construct cue words, input the cue words into a large language model, and use the large language model to generate attribute text sets for each category in the image dataset. Each category's attribute text set includes multiple attribute texts, which are connected by concatenation symbols. The specific method for constructing a training sample set based on attribute text is as follows: Summarize the attribute text sets of all categories, and remove duplicate attribute text to obtain the following: The training sample set of attribute text.

3. The visual language model image classification method based on attribute anchor cue word fine-tuning according to claim 2, characterized in that, The attribute compression module includes a first normalization layer, a first fully connected layer, a GELU activation layer, and a second fully connected layer. The specific method for training the attribute compression module using the training sample set is as follows: The batches will include The training sample set of attribute texts is input in parallel into the text encoder of the pre-trained visual language model, and the parameters of the text encoder are frozen. Each batch of attribute texts is encoded into a token embedding sequence by a text segmenter and fed into the text encoder to extract the text features of the attribute texts. The text features of the attribute texts are used as the teacher text features. The token embedding sequence is compressed into a soft-attribute token embedding sequence of length K based on the attribute compression module, thus obtaining a soft-attribute token tensor; The soft attribute token tensor is input into the text encoder to obtain the feature vector after attribute compression. The feature vector after attribute compression is then processed... Norm normalization operation yields student text features; A bidirectional main loss function is constructed based on teacher and student text features, and the attribute compression module is trained to obtain the trained attribute compression module. The bidirectional main loss function includes the total loss function from student to teacher and the total loss function from teacher to student.

4. The visual language model image classification method based on attribute anchor cue word fine-tuning according to claim 3, characterized in that, The specific method for constructing a static attribute pool based on the trained attribute compression module is as follows: Load the trained attribute compression module and text encoder; freeze the parameters of the trained attribute compression module and text encoder completely without performing gradient updates; Traverse all categories in the image dataset, input the attribute text set corresponding to each category into the frozen attribute compression module in parallel, and calculate the soft attribute token tensor corresponding to each category. Each category's corresponding soft attribute token tensor is stored using the category as the index, constructing a category-level static attribute pool, where the category... The soft property token tensor is .

5. The image classification method based on attribute anchor cue word fine-tuning of a visual language model according to claim 4, characterized in that, The cross-attention aggregation module includes a cross-attention mechanism, layer normalization, and a feedforward neural network; The cross-attention aggregation module is used to aggregate the soft attribute token tensors in the static attribute pool. and category name embedding Aggregation, resulting in aggregated embeddings The specific method is as follows: Initialize a set of random, learnable vectors as query vectors. ; For category Retrieve the corresponding soft attribute token tensor from the static attribute pool. , the soft attribute token tensor and category name embedding Concatenate the sequences to build a key-value pool. ; Utilizing a multi-head attention mechanism, based on the query vector and key-value pool The attention output is calculated and processed through layer normalization and a feedforward neural network to obtain the aggregated embedding. .

6. The visual language model image classification method based on attribute anchor cue word fine-tuning according to claim 1, characterized in that, For cross-attention aggregation modules and learnable soft prefixes The specific method for fine-tuning parameters is as follows: Extracting text feature vectors based on mixed prompt words ; For each category in the image dataset, 80 standard handwritten templates provided by ImageNet are obtained. The category name of each category is filled into the standard handwritten template, and then input into the text encoder to extract features. Each category yields 80 standard handwritten template feature vectors. Calculate the mean of the feature vectors of 80 handwritten standard templates for each category, and then perform... Normalization yields the semantic anchor feature vector for each category. ; Based on text feature vectors and semantic anchor feature vector Construct a joint objective function and fine-tune the parameters of the cross-attention aggregation module and the soft cue prefix; During training, only the parameters of the cross-attention aggregation module and the soft prefix are updated until the training reaches the preset number of iterations or the loss function converges, thus obtaining the final trained parameters of the cross-attention aggregation module and the optimized soft prefix. During the inference phase, the trained cross-attention aggregation module is used to generate aggregated embeddings. And concatenate it with the category name embedding and optimized soft prefix to generate mixed prompt words; The mixed prompts and the image dataset to be identified are input into a pre-trained visual language model to identify the category of each image in the image dataset.

7. The visual language model image classification method based on attribute anchor cue word fine-tuning according to claim 6, characterized in that, The joint objective function includes contrastive loss and semantic regularization loss; the contrastive loss includes image-to-text loss and text-to-image loss; the strength of the semantic regularization loss is adjusted by regularization weights; The regularization weights employ a linear warm-up strategy, increasing linearly from 0 to a preset value in the first few training rounds and then remaining unchanged thereafter.

8. An electronic device, comprising: One or more processors, and a memory for storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the visual language model image classification method based on attribute anchor cue word fine-tuning as described in any one of claims 1-7.

9. A computer-readable storage medium storing executable instructions, which, when executed, cause a processor to perform the image classification method of a visual language model based on attribute anchor cue word fine-tuning as described in any one of claims 1-7.

10. A computer program product comprising a computer program or instructions which, when executed by a processor, implement the visual language model image classification method based on attribute anchor cue word fine-tuning as described in any one of claims 1-7.

Images