Fine-grained image classification method based on text enhancement

Abstract

The invention provides a fine-grained image classification method based on text enhancement, and the method comprises the steps: constructing a fine-grained image description data set, and carrying out the fine adjustment of a pre-trained multi-modal large model through the fine-grained image description data set, so as to obtain a multi-modal large model with the capability of generating the description of a fine-grained image; extracting an image feature and a text feature of the input image by using the multi-modal large model after fine tuning, and fusing the text feature and the image feature through an attention mechanism to obtain a final classification feature; and inputting the final classification features into a classifier to obtain an image classification result output by the classifier. Therefore, the recognition capability and the classification performance of the model on the subtle visual differences can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a fine-grained image classification method based on text enhancement. BACKGROUND

[0002] Fine-grained image classification is a classification problem in computer vision, aiming to distinguish classes with high similarity. Unlike traditional image classification, fine-grained image classification needs to distinguish specific subcategories (such as different species of birds, different brands and models of cars) within the same large category (such as birds, cars, flowers, etc.). This technology is widely used in biological species identification, industrial quality detection, etc. In the fields of ecological protection, food safety, intelligent transportation, etc., it learns subtle visual differences through deep learning models to achieve high-precision automatic identification, providing important technical support for scientific research, industry and daily life.

[0003] Currently, there are representative fine-grained image classification methods:

[0004] 1) Fine-grained image classification method based on explicit regions and attributes. It extracts local features to assist fine-grained image classification by explicitly detecting key regions in images or using attribute labeling information. The core idea of this method is to use additional labeling information (such as attributes or key regions) to enhance the model's ability to capture fine-grained features, thereby better distinguishing similar classes. The advantage of this method is that it can guide model learning through explicit attribute information, but it relies on high-quality labeled data, which is costly.

[0005] 2) Fine-grained image classification method based on attention mechanism. It automatically focuses on key regions in images through deep learning models, reducing the dependence on manual labeling, and combining global and local features for classification. The core of this method is to use attention mechanism to dynamically select important regions in images, thereby improving the model's ability to capture fine-grained features. The advantage of this method is that it can automatically learn important regions in images, but its performance is greatly affected by the design of attention mechanism, and it lacks understanding of multi-modal information, and performance improvement is bottlenecked.

[0006] 3) Classification method based on contrast learning. It mainly learns the subtle differences between similar samples to construct discriminative feature representations. The core idea is to pull similar samples closer in feature space and push different class samples further apart, and optimize the model learning class boundaries through contrast loss function. This method can effectively learn key visual features to distinguish similar classes under limited labeled data, significantly improving the accuracy and generalization ability of fine-grained image classification. However, contrast learning relies on the design of positive and negative sample selection strategy, which is difficult, and the algorithm has high computational complexity.

[0007] Therefore, there is an urgent need for a method that can effectively solve the above technical problems. SUMMARY

[0008] In order to solve the problems of the prior art, the present application provides a fine-grained image classification method based on text enhancement, which uses text information to enhance image feature representation and improve the performance of the model in the classification task.

[0009] In order to achieve the above technical effects, the present application provides a fine-grained image classification method based on text enhancement, comprising the steps of:

[0010] Constructing a fine-grained image description dataset and fine-tuning a pre-trained multi-modal large model using the fine-grained image description dataset to obtain a multi-modal large model capable of generating fine-grained image descriptions;

[0011] Using the fine-tuned multi-modal large model to extract image features and text features of an input image, and fusing the text features and the image features through an attention mechanism to obtain final classification features;

[0012] Inputting the final classification features into a classifier to obtain an image classification result output by the classifier.

[0013] Further, the construction of the fine-grained image description dataset specifically includes:

[0014] Obtaining a dictionary meaning and an individual image description of a target category;

[0015] Using a language model to fuse the dictionary meaning and the individual image description to generate a fine-grained image description for each image;

[0016] Based on the generated fine-grained image description for each image, a fine-grained image description dataset is constructed.

[0017] Further, in the process of fine-tuning the pre-trained multi-modal large model, the visual encoder and the large language model of the multi-modal large model are frozen, and only the query conversion module is fine-tuned.

[0018] Further, the query conversion module is a Q-Former module.

[0019] Further, using the fine-tuned multi-modal large model to extract image features and text features of an input image, and fusing the text features and the image features through an attention mechanism to obtain the final classification features, specifically includes:

[0020] Using a visual encoder to extract global visual features of an input image;

[0021] The query representation containing visual information is input into a large language model to generate a text feature;

[0022] The query representation containing visual information is input into a large language model to generate a text feature;

[0023] The feature dimension of the query representation containing visual information is adjusted to be consistent with the text feature dimension using a fully connected layer;

[0024] An enhanced query representation is generated using a multi-head attention mechanism with the adjusted query representation as the query, the text feature as the key and value, and the attention output combined with an enhancement coefficient.

[0025] Further, the enhanced query representation is generated by the following formula:

[0026] ;

[0027] Wherein, is the enhanced query feature, is the enhancement coefficient, is the query representation containing visual information consistent with the dimension of the text feature; is the text feature output by the last layer of the large language model; represents the multi-head attention calculation with as the query, as the key and value.

[0028] Further, the image feature and text feature of the input image are extracted using the fine-tuned multi-modal large model, and the text feature and the image feature are fused through an attention mechanism to obtain the final classification feature, which further includes:

[0029] The similarity between each enhanced query representation and the average-pooled text feature is calculated;

[0030] The query feature with the highest similarity is selected as the key query feature;

[0031] The key query feature and the average of all enhanced query representations are weighted and summed using a weight coefficient to generate the final classification feature.

[0032] Further, the final classification feature is generated by the following formula:

[0033] ;

[0034] Wherein, is the key query feature, is the weight coefficient, M is the average value of all enhanced query representations The number of query vectors.

[0035] Further, the classifier includes a fully connected layer and a Softmax function, and the classification loss is calculated by a cross-entropy loss function.

[0036] Further, the formula for calculating the cross-entropy loss function is:

[0037] ;

[0038] where C is the total number of categories, is the value of the one-hot encoding of the true class label on the c-th category, is the probability that the model predicts the c-th category.

[0039] The fine-grained image classification method based on text enhancement provided by the present application combines the dictionary meaning of the target category with the image description to construct a fine-grained image description data set, and fine-tunes a multi-modal large model on this data set to enhance the ability of the multi-modal large model to generate fine-grained descriptions. Subsequently, based on the image features extracted by the fine-tuned multi-modal large model and the generated text features, the present application also realizes the enhancement of the text features to the image features through the attention mechanism. In this way, the present application provides the model with more rich semantic information, realizes the effective enhancement of the text features to the image features, improves the recognition ability of the model to fine-grained features through cross-modal information complementation, and verifies that the present application can effectively improve the performance of the model in the fine-grained image classification task. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 The step flowchart of the fine-grained image classification method based on text enhancement provided by an embodiment of the present application is shown in the figure;

[0041] Figure 2 The flowchart of constructing a fine-grained image description data set by using the fine-grained image classification method based on text enhancement provided by an embodiment of the present application is shown in the figure;

[0042] Figure 3 The flowchart of image classification by using the fine-grained image classification method based on text enhancement provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0044] It is to be understood that the references used herein to "one embodiment", "an embodiment", "example embodiment", etc. do not necessarily all refer to the same embodiment, although they can. Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. The terminology used herein is for the purpose of describing the embodiments only and is not intended to be limiting. It will be understood that the use of "including", "comprising", "having" and "with" and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. The use of "consisting of" and variations thereof is meant to exclude items not specifically listed.

[0045] In addition, certain terms used in the specification and following claims are to be construed in accordance with the knowledge of one skilled in the art. The terminology used herein is for the purpose of describing only the particular embodiments and is not intended to be limiting. The use of the terms "including", "comprising", "having" and "with" along with their derivatives, are meant to encompass the items listed thereafter and equivalents thereof as well as additional items. The use of the terms "consisting of" and "consisting essentially of" are meant to exclude items not specifically listed.

[0046] In the research of fine-grained image classification, it is found that the existing methods lack the guidance of text information when performing fine-grained image classification tasks, or only use image modal information, or use class labels with less information as text information supplement, and experiments verify that supplementing text information can effectively improve the performance of fine-grained image classification. In view of this, the fine-grained image classification method based on text enhancement provided by the present application uses the image text description generated by the multi-modal large model as auxiliary information to enhance the image feature representation and improve the classification performance.

[0047] The fine-grained image classification method based on text enhancement provided by the embodiments of the present application will be further described in detail below in combination with the accompanying drawings of the specification.

[0048] Figure 1 The fine-grained image classification method based on text enhancement provided by the embodiments of the present application will be further described in detail below in combination with the accompanying drawings of the specification.

[0049] S101: Construct a fine-grained image description data set, and use the fine-grained image description data set to fine-tune a pre-trained multi-modal large model to obtain a multi-modal large model with the ability to generate fine-grained image descriptions.

[0050] In the process of fine-tuning the pre-trained multi-modal large model, the visual encoder and the large language model of the multi-modal large model are frozen, and only the query conversion module is fine-tuned, and the query conversion module is a Q-Former module.

[0051] In a specific embodiment, the constructing the fine-grained image description dataset specifically comprises: obtaining a dictionary meaning of a target category and an individual image description; fusing the dictionary meaning and the individual image description using a language model to generate a fine-grained image description of each image; and constructing the fine-grained image description dataset based on the generated fine-grained image description of each image.

[0052] The following embodiment will be described by taking an experiment on the food dataset FruitVeg-81 as an example.

[0053] In order to obtain fine-grained food image descriptions, the embodiment fuses food category dictionary meanings and individual image descriptions, and the specific process is as shown in Figure 2 The category dictionary meanings are derived from mainstream encyclopedias, and the individual image descriptions are generated by a pre-trained multi-modal large model. After obtaining the dictionary meaning of the food category and the description of the single image, the two are further input into the pre-trained large language model to generate fine-grained image descriptions using the powerful language generation capability of the large language model, and finally the fine-grained image description dataset is obtained.

[0054] The pre-trained multi-modal large model is fine-tuned on the constructed fine-grained image description dataset. During the fine-tuning process, the visual encoder and the large language model in the multi-modal large model architecture are frozen, and only the Q-Former module is fine-tuned, thereby achieving efficient alignment of visual features and text features. The fine-tuned multi-modal large model has the ability to generate fine-grained image descriptions.

[0055] S102: using the fine-tuned multi-modal large model to extract image features and text features of an input image, and fusing the text features and the image features through an attention mechanism to obtain final classification features.

[0056] In a specific embodiment, step S102 comprises: using a visual encoder to extract global visual features of an input image; using a learnable query tensor to interact with the global visual features through a query transformation module to generate a query representation containing visual information; inputting the query representation containing visual information into a large language model to generate text features; using a fully connected layer to adjust the feature dimension of the query representation containing visual information to be consistent with the feature dimension of the text features; using a multi-head attention mechanism to take the adjusted query representation as a query, and the text features as a key and a value, to calculate an attention output, and combine an enhancement coefficient to generate an enhanced query representation.

[0057] Further, step S102 further comprises: calculating the similarity between each enhanced query representation and the average-pooled text features; selecting the query feature with the highest similarity as a key query feature; and using a weight coefficient to weight and sum the key query feature and the average of all enhanced query representations to generate final classification features.

[0058] S103: input the final classification features into a classifier to obtain an image classification result output by the classifier. The classifier includes a full connection layer and a Softmax function, and a classification loss is calculated by a cross-entropy loss function.

[0059] The specific process of performing a classification task by the embodiment using a text enhanced image is shown in FIG. 2. First, an input image I extracts global visual features F through a pre-trained visual encoder (such as a Vision Transformer, ViT). Then, a Query tensor containing 32 initialized learnable query vectors is used to interact with the visual features F through a Q-Former module to generate a query representation containing visual information Figure 3 . Specifically, the output of the Q-Former is obtained by updating the Query tensor through a multi-head self-attention mechanism and a feedforward network.

[0060] The query representation containing visual information is input into a pre-trained large language model to generate a text feature T e . The large language model generates a high-dimensional representation of the text description related to the image through an autoregressive generation process, combining the Query representation and the knowledge base of the language model.

[0061] In order to keep the query feature and the text feature T e consistent in the feature dimension, the embodiment introduces a full connection layer before the large language model, which maps to the same feature dimension as T e , obtaining :

[0062] ;

[0063] wherein is the query representation containing visual information; is the weight matrix of the full connection layer; is the bias vector; is the query feature after changing the dimension, B is the block size, M is the number of queries, D I and D T are the feature dimensions of the image and the text respectively.

[0064] In order to realize the effective interaction of the query feature and the text feature, the query feature is enhanced using a multi-head attention mechanism. In order to maintain the stability of the query feature during the enhancement process and avoid over-adjustment, an enhancement coefficient​ This is used to control the magnitude of the enhancement. Specifically, this embodiment will... Set it to 0.1 to achieve moderate feature enhancement.

[0065] In multi-head attention mechanisms, query features With text features T e Interact to generate enhanced query representations. :

[0066] ;

[0067] in These are enhanced query features. These are the text features output by the last layer of the large language model; Indicates For querying, T e Multi-head attention computation for keys and values.

[0068] In order to obtain First, the learnable query tensor Q is initialized, and then it is obtained by interacting with the features output by the visual encoder through Q-Former. ,at last A linear projection layer is used to obtain the same dimension as the text features. .

[0069] To select the query feature that best reflects the category from all query features for the classification task, this embodiment calculates the similarity between each query feature and the average pooled text features, and selects the query feature most similar to the text. Meanwhile, to avoid ignoring information from other query features, weighting coefficients were introduced. All query features are weighted and pooled to generate the final classification features. Finally, The input classification header is used for classification, and the classification loss is calculated using the cross-entropy loss function.

[0070] Calculate each query feature With text features T e The similarity is calculated using a dot product to select the query that best reflects the category characteristics.

[0071] ;

[0072] in It is the similarity between the i-th query feature and the text feature.

[0073] By comparing all similarities Select the query feature with the highest similarity as :

[0074] ;

[0075] To avoid ignoring information from other query features, this embodiment introduces a weighting coefficient. All query features are weighted to generate the final classification features. :

[0076] ;

[0077] in, It is the average value of all query features.

[0078] Final classification features The input is classified by the classification head to generate category predictions. :

[0079] ;

[0080] in and These are the weights and biases of the classification heads; It is the probability distribution for category prediction.

[0081] The classification loss is calculated using the cross-entropy loss function, as shown in the following formula:

[0082] ;

[0083] Where C is the number of categories; It is the one-hot encoding of the true category; It is the probability of predicting the category.

[0084] During the testing phase, the target image is input into the model, and the enhanced image features are obtained through the above steps. These enhanced features are then input into the trained classification head to obtain the classification result.

[0085] Through the above process, the query features most similar to the text features were selected from all query features. At the same time, the final classification features are generated by combining information from all query features. This design highlights key query features while preserving global information, thereby improving the performance of classification tasks.

[0086] In summary, the text-enhanced fine-grained image classification method of this invention constructs a fine-grained image description dataset by combining the dictionary meaning of the target category with image descriptions. A multimodal large model is then fine-tuned on this dataset to enhance its ability to generate fine-grained descriptions. Subsequently, based on the image features extracted from the fine-tuned multimodal large model and the generated text features, this invention further enhances image features through an attention mechanism. Thus, this invention provides the model with richer semantic information, effectively enhances image features through text features, and improves the model's ability to recognize fine-grained features through cross-modal information complementarity. Experiments verify that this invention can effectively improve the performance of the model in fine-grained image classification tasks.

[0087] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0088] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A fine-grained image classification method based on text enhancement, characterized in that, The method comprises the steps of: constructing a fine-grained image description dataset and fine-tuning a pre-trained multi-modal large model using the fine-grained image description dataset to obtain a multi-modal large model capable of generating fine-grained image descriptions; extracting image features and text features of an input image using the fine-tuned multi-modal large model, and fusing the text features and the image features through an attention mechanism to obtain final classification features; inputting the final classification features into a classifier to obtain an image classification result output by the classifier.

2. The method of claim 1, wherein, The construction of the fine-grained image description dataset specifically comprises: obtaining a dictionary meaning and an individual image description of a target category; fusing the dictionary meaning and the individual image description using a language model to generate a fine-grained image description for each image; constructing a fine-grained image description dataset based on the generated fine-grained image description for each image.

3. The method of claim 1, wherein, In the process of fine-tuning the pre-trained multi-modal large model, the visual encoder and the large language model of the multi-modal large model are frozen, and only the query conversion module is fine-tuned.

4. The method of claim 3, wherein, The query conversion module is a Q-Former module.

5. The method of claim 1, wherein, The use of the fine-tuned multi-modal large model to extract image features and text features of an input image and to fuse the text features and the image features through an attention mechanism to obtain final classification features specifically comprises: extracting global visual features of the input image using a visual encoder; generating a query representation containing visual information by interacting the global visual features with the query conversion module using a learnable query tensor; inputting the query representation containing visual information into a large language model to generate a text feature; adjusting the feature dimension of the query representation containing visual information to be consistent with the dimension of the text feature using a fully connected layer; using the adjusted query representation as a query, the text feature as a key and a value, and a multi-head attention mechanism to calculate an attention output and generate an enhanced query representation in combination with an enhancement coefficient.

6. The method of claim 5, wherein, The enhanced query representation is generated by the following formula: ； wherein, is the enhanced query feature, is the enhancement coefficient, is the query representation containing visual information consistent with the feature dimension of the text feature; is the text feature of the last layer output of the large language model; represents that is the query, is the multi-head attention calculation of the key and value.

7. The method of claim 5, wherein, The use of the fine-tuned multi-modal large model to extract image features and text features of an input image and to fuse the text features and the image features through an attention mechanism to obtain the final classification features further comprises: calculating the similarity between each enhanced query representation and the average-pooled text feature; selecting the query feature with the highest similarity as the key query feature; using a weight coefficient to weight and sum the average of the key query feature and all enhanced query representations to generate the final classification feature.

8. The method of claim 7, wherein, The final classification feature is generated by the following formula: ； wherein, is the key query feature, is a weight coefficient, is the average of all enhanced query representations, M is the enhanced query feature the number of query vectors.

9. The method of claim 1, wherein, The classifier comprises a fully connected layer and a Softmax function, and the classification loss is calculated by a cross-entropy loss function.

10. The method of claim 1, wherein, The calculation formula of the cross-entropy loss function is: ； where C is the total number of classes, is the value of the one-hot encoding of the true class label on the c-th class, is the probability that the model predicts the c-th class.

Images