Fine-grained image classification method based on large model enhancement

Abstract

The invention discloses a fine-grained image classification method based on large model enhancement, which guides a model to learn a specific judgment mode of a task by directly introducing prior knowledge. A series of image descriptions are generated by performing question and answer interaction with a multi-modal large language model (MLLM). In order to filter illusion information and redundant content existing in description, a dual-guide text feature optimization module is introduced, and the quality of text features is improved through task-guided feature selection and similarity-guided feature pruning. And finally, adopting a multilayer network structure based on an attention mechanism to realize vision-language fusion for final classification prediction.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a fine-grained image classification method based on large model enhancement. Background Technology

[0002] Fine-grained image classification (FGIC) aims to train models to fit complex rules developed by experts using interdisciplinary knowledge, thereby distinguishing categories of objects with extremely similar appearances. With the continuous expansion of application scenarios, especially in fields such as biodiversity conservation, industrial monitoring, and medical image analysis, the demand for FGIC is becoming increasingly prominent. For example, through offline recording with fixed-point cameras, FGIC can assist in monitoring the distribution and activity of specific species or objects, supporting research and applications. Existing FGIC methods are mainly divided into two categories: models based on convolutional neural networks (CNNs) and models based on Transformers. In recent years, with the advantages of Transformer-based methods in capturing global features and modeling long-range dependencies, they have gradually become the mainstream backbone models in FGIC tasks. However, these models, which rely solely on visual information, are limited by the size of the dataset and are difficult to optimize sufficiently, thus limiting their potential in FGIC. To address this, MP-FGVC introduces learnable text tokens and utilizes CLIP's cross-modal description capabilities to fit subclassing rules, significantly improving the performance of fine-grained classification. However, these methods still essentially rely on the model spontaneously learning complex classification rules developed by experts during training. While this highly data-driven "black box" paradigm can extract certain discriminative features, it still struggles to achieve robust recognition and generalization in complex scenarios (such as complex backgrounds, posture changes, or bird molting).

[0003] Therefore, the FGIC task was re-examined, such as Figure 1 As shown in (a), a phenomenon of "small inter-class differences and large intra-class differences" was observed in FGIC: there are only very small differences between two different bird classes (a&c, b&d). Meanwhile, two birds belonging to the same class (a&b, c&d) exhibit significant visual differences. These phenomena arise because fine-grained classification does not rely solely on appearance features, but is based on complex rules that integrate multi-dimensional knowledge (such as morphology, ecology, and behavior). However, existing models often attempt to fit these high-level, multi-dimensional discrimination rules using only limited fine-grained data, resulting in limited generalization ability and difficulty in overcoming performance bottlenecks. This raises the question: if these rules were explicitly provided directly to the model, could the optimization process be simplified and the ability to extract discriminative features enhanced? Summary of the Invention

[0004] Based on the above analysis, this invention proposes a fine-grained image classification method based on large model enhancement. This method utilizes a multimodal large language model (MLLM) via question-and-answer, directly injecting text-based expert prior knowledge into the FGIC model. Considering the potential for illusory and redundant information in the MLLM output, a dual-guided text optimization module is introduced. This module selects more discriminative text features through task-guided feature selection and similarity-guided feature pruning. Finally, an attention-based network architecture is employed to fuse visual features with the optimized text features, thereby performing the final fine-grained classification prediction.

[0005] The fine-grained image classification method based on large model enhancement disclosed in this invention specifically includes the following steps:

[0006] Step 1: A visual description generation method was constructed, which interacts with MLLM in a question-and-answer manner to generate multi-level visual descriptions for each input image.

[0007] Step 2: For the multi-level visual description of the input image, it is processed into text features by a text encoder; the input image is subjected to multi-level image feature extraction through a backbone model;

[0008] Step 3: Construct and train the network model, which includes three parts: visual description generation, dual-guided text feature refinement, and visual-language fusion. Except for the visual description generation part, which does not require training, the other parts of the network model are learnable. Use the trained model to process image embedding to achieve high-precision image recognition. Construct a network model based on the attention mechanism and input data into the network model for end-to-end training.

[0009] Step 4: Fine-grained image classification. Input fine-grained images into the trained network model to perform high-precision image classification.

[0010] The specific details of each step are as follows. For ease of understanding, we will take fine-grained bird classification as an example:

[0011] The visual description generation method in step 1 first constructs three layers of key prompts to facilitate understanding and generation of visual descriptions consistent with expert knowledge for multimodal large models: "whole image," "pose," and "discriminative features." Then, these key prompts are embedded into carefully designed templates to ultimately form three layers of prompt sentences. Describe the entire image in one sentence. Describe the bird's posture in the picture in one sentence; "Describe the discriminative features of the bird in the image in one sentence." (For the input bird image) These prompts, through a question-and-answer format, guide MLLM to generate multi-layered visual descriptions containing bird expert-discriminative rules. ,in .

[0012] Step 2 extracts text features containing expert knowledge from the multi-layer visual description of the input image using a text encoder. Simultaneously, a visual backbone model is used to extract multi-level visual features from the input image. .

[0013] The dual-guided text feature refinement part described in step 3 constructs a corresponding dual-guided text feature refinement module (DTR) for each level of visual description. In this module, adversarial learning is first used for feature alignment, making the text feature space and visual feature space closer. Then, a task-guided text feature selector and a similarity-guided text feature pruner are used to effectively refine illusions and redundant information in prior knowledge, improving the guiding effect of text features on the visual model. Specifically:

[0014] Feature Alignment: The feature alignment module consists of two independent multilayer perceptrons (MLPs) and a modality classifier. Specifically, the two MLPs are used to align visual features... Text features Mapping to the same dimension yields... and ,in and They represent the first Visual and textual layers quantity, The projected feature dimensions. Each MLP consists of a linear layer and a... The system consists of layers that project the input features into a unified 512-dimensional space. The modality classifier comprises two linear layers and one... The activation function first maps the input from 512 dimensions to 256 dimensions, then outputs a 2D vector for modality discrimination. During training, the modality classifier aims to distinguish visual and textual features as much as possible, while the MLP generates modality-indistinguishable features to confuse the classifier, thus achieving cross-modality alignment. To this end, an adversarial mechanism is introduced between the MLP and the modality classifier. During backpropagation, a gradient inversion layer (GRL) is used to invert the gradient of the modality classifier, thereby constructing a minimax game that enables the network to output aligned cross-modality features. and .

[0015] Task-Guided Text Feature Selector: Visual descriptions generated by MLLM often contain illusions or invalid information, which can lead to biased decisions in FGIC tasks that require subtle details. Therefore, a Task-Guided Text Feature Selector (TTS) is introduced, supervised by the FGIC task. It evaluates bird rule information using learnable scoring tokens to extract discriminative text representations for classification decisions. Specifically, firstly, in the aligned text features... Introducing learnable ratings This is used to evaluate the importance of each text token. Then, and Features obtained by splicing And map it to the query through a linear transformation. ,key Sum Form. By calculating query-key dot product similarity and applying... Normalization yields the attention weight matrix. :

[0016]

[0017] exist Extract the rating token from the middle. Text token Attention distribution vector And retain based on its score Important text tokens to construct refined features :

[0018]

[0019] in, Indicates based on vector (only after use) (Scores for each text token) from features Select the highest score Each token. To ensure To improve the effectiveness of the scoring and further optimize its representation, specifically, the following will be implemented: AND value vector Multiply the results to aggregate key information relevant to the rating tokens. Then, perform a linear transformation. After processing, compared with the original rating token Perform a residual join to obtain the updated representation of the score token:

[0020]

[0021] Finally, the optimized rating token will be... Input to category header Furthermore, supervised training is performed using cross-entropy loss to remove task-irrelevant or illusory information, thereby improving the guiding effect of text features in FGIC.

[0022] Similarity-guided text feature pruning: In this invention, it is hoped that text can provide rich prior knowledge, guiding visual models to focus on subtle differences that are difficult for them to fit on their own. However, if there is too much overlap between text and visual features, not only will it fail to guide the model to focus on these subtle differences, but it may also cause the model to assign more weight to some salient features, resulting in reduced model performance and generalization. Therefore, a similarity-guided text feature pruning (STP) is embedded in the task-guided text feature selector. Through similarity analysis between visual and textual features, semantically redundant text tokens are identified and pruned. Specifically, firstly, for visual features... and text features Perform a linear transformation and project it onto the query. s and keys Space, then calculate the scaled dot product similarity between the visual token and the text token:

[0023]

[0024] in, This represents the similarity score matrix. Further, the average similarity score between all visual tokens and text tokens is calculated to obtain the score vector.

[0025]

[0026] exist A higher score for a text token indicates that it receives significant attention across multiple visual tokens, meaning its semantic information may highly overlap with or be overly generic with visual features. Therefore, according to... Sort in descending order and remove the highest-scoring items. Text token, to obtain the final refined text features :

[0027]

[0028] in, Indicates based on the rating vector From the characteristics Remove the highest scorer from the list. item.

[0029] The visual-language fusion component described in step 3 consists of a visual-language fusion network. This network enhances intra-modal and inter-modal interactions through a multimodal fusion module (MFM) and aggregates multi-scale features in a top-down manner. Specifically, in the MFM, refined text features from the same layer are... and visual features The features are then spliced ​​together to form preliminary multimodal features. Subsequently, a self-attention mechanism was used to fully extract relevant information within each modality, resulting in optimized multimodal features. Then, the enhanced visual features are extracted from them. Next, As a query Text features As a key Sum By leveraging the cross-attention mechanism to further integrate information between modalities, and fully utilizing the prior knowledge contained in the text features to enhance the localization of key discriminative regions in the visual features, this process can be represented as:

[0030]

[0031]

[0032]

[0033]

[0034] in, This indicates the extraction of the former from multimodal features. The visual portion of each location. The dimension representing the key is used to scale the attention score. Finally, a top-down architecture is adopted, using 3×3 convolutions to progressively downsample and fuse multi-scale features, transferring and aggregating prior knowledge from each level to the final visual features. .

[0035] Will After processing by the classification head, fine-grained image classification and prediction are performed.

[0036] This method proposes a fine-grained image classification framework based on large model enhancement. This framework leverages the high generalization ability of MLLM (Multi-Level Model) to introduce structured prior knowledge, thereby guiding the visual model to make effective decisions. Specifically, firstly, a visual description generation method is constructed that interacts with MLLM via a question-and-answer approach to generate multi-level visual descriptions. Secondly, to address the potential for redundancy and illusory information in MLLM-generated descriptions, a dual-guided text feature refinement module (DTR) is introduced, incorporating a task-guided text feature selector (TTS) and a similarity-guided text feature pruner (STP) to process text features. Finally, an attention-based network is employed to fuse visual and linguistic features for the final prediction. Attached Figure Description

[0037] Figure 1 This is a schematic diagram illustrating the phenomena present in the task addressed by the present invention.

[0038] Figure 2 The flowchart of the entire algorithm proposed in this invention.

[0039] Figure 3 This is a diagram of the dual-guided text feature refinement module architecture proposed in this invention.

[0040] Figure 4 This is an architecture diagram of the feature alignment module proposed in this invention.

[0041] Figure 5 This is an architecture diagram of the multimodal fusion module proposed in this invention. Detailed Implementation

[0042] The purpose of this invention is to propose a fine-grained image classification method based on large model enhancement. This invention is designed specifically for the characteristics of fine-grained data and performs fine-grained image classification on a trained deep learning model.

[0043] The following detailed description of the technical solution provided by the present invention will be based on fine-grained bird classification as a specific embodiment. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0044] The workflow of this invention is divided into a training phase and a prediction phase. In the training phase, learnable parameters in the dual-guided text feature refinement and visual-language fusion components are trained using training set data, enabling the model to learn sufficient classification knowledge from a large number of fine-grained classification samples. In the prediction phase, based on the trained model parameters, different fine-grained images are classified.

[0045] Benefiting from the rapid development of MLLM, this invention aims to leverage the high generalization ability of MLLM to generate corresponding text-based prior knowledge for each sample. To this end, this invention proposes a visual description generation method, such as... Figure 2 As shown, firstly, to facilitate understanding and generation of visual descriptions consistent with expert knowledge by the multimodal large model, a three-layered set of key prompts is constructed: "whole image," "pose," and "discriminative features." Then, these key prompts are embedded into carefully designed templates to ultimately form a three-layered prompt sentence. Describe the entire image in one sentence. Describe the bird's posture in the picture in one sentence; "Describe the discriminative features of the bird in the image in one sentence." (For the input bird image) These prompts, through a question-and-answer format, guide MLLM to generate multi-layered visual descriptions containing bird expert-discriminative rules. ,in The MLLM used is gemini-2.0-flash.

[0046]

[0046] The multi-layer visual description of the input image is extracted using CLIP's text encoder to extract text features containing expert knowledge. Simultaneously, the Swing Transformer visual backbone model is used to extract multi-level visual features from the input image. .

[0047] In obtaining multi-layer text features of the input image and visual features Afterwards, according to Figure 2 , 3 4.5 Construct the algorithm network of this invention. The network model includes three parts: visual description generation, dual-guided text feature refinement, and visual-language fusion. The data processing flow of the entire algorithm is as follows: Figure 2 As shown:

[0048] Dual-guided text feature refinement: refining multi-layer text features from the input image and visual features The input is fed into a dual-guided text feature refinement module (DTR) at each level. This module first uses adversarial learning for feature alignment, bringing the text feature space closer to the visual feature space. Then, a task-guided text feature selector and a similarity-guided text feature pruner are used to effectively refine illusions and redundant information in prior knowledge, improving the guiding effect of text features on the visual model. Specifically:

[0049] Feature alignment: The feature alignment module consists of two independent multilayer perceptrons (MLPs) and a modality classifier, such as... Figure 3 As shown in (a). Specifically, the two MLPs are used to process visual features. Text features Mapping to the same dimension yields... and ,in and They represent the first Visual and textual layers quantity, The projected feature dimensions. Each MLP consists of a linear layer and a... The layers project the input features onto a unified 512-dimensional space, such as... Figure 4 As shown. The modality classifier consists of two linear layers and one... The activation function is constructed by first mapping the input from 512 dimensions to 256 dimensions, and then outputting a 2D vector for mode discrimination, such as... Figure 4 As shown, during training, the modality classifier aims to distinguish visual and textual features as much as possible, while the MLP generates modality-indistinguishable features to confuse the classifier, thereby achieving cross-modality alignment. To this end, an adversarial mechanism is introduced between the MLP and the modality classifier. During backpropagation, a gradient inversion layer (GRL) is used to invert the gradient of the modality classifier, thus constructing a minimax game that enables the network to output aligned cross-modality features. and .

[0050] Task-Guided Text Feature Selector: Visual descriptions generated by MLLM often contain illusions or invalid information, which can lead to biased decisions in FGIC tasks that require subtle details. Therefore, a Task-Guided Text Feature Selector (TTS) is introduced, supervised by the FGIC task. It evaluates prior knowledge using learnable scoring tokens to extract discriminative text representations for classification decisions. Specifically, firstly, in the aligned text features... Introducing learnable ratings This is used to evaluate the importance of each text token. Then, and Features obtained by splicing And map it to the query through a linear transformation. ,key Sum Form. By calculating query-key dot product similarity and applying... Normalization yields the attention weight matrix. :

[0051]

[0052] exist Extract the rating token from the middle. Text token Attention distribution vector And retain based on its score Important text tokens to construct refined features :

[0053]

[0054] in, Indicates based on vector (only after use) (Scores for each text token) from features Select the highest score Each token, Set to 15. To ensure... To improve the effectiveness of the scoring and further optimize its representation, specifically, the following will be implemented: AND value vector Multiply the results to aggregate key information relevant to the rating tokens. Then, perform a linear transformation. After processing, compared with the original rating token Perform a residual join to obtain the updated representation of the score token:

[0055]

[0056] Finally, the optimized rating token will be... Input to category header Furthermore, supervised training is performed using cross-entropy loss to remove task-irrelevant or illusory information, thereby improving the guiding effect of text features in FGIC.

[0057] Similarity-guided text feature pruning: In this invention, it is hoped that text can provide rich prior knowledge, guiding visual models to focus on subtle differences that are difficult for them to fit on their own. However, if there is too much overlap between text and visual features, not only will it fail to guide the model to focus on these subtle differences, but it may also cause the model to assign more weight to some salient features, resulting in reduced model performance and generalization. Therefore, a similarity-guided text feature pruning (STP) is embedded in the task-guided text feature selector. Through similarity analysis between visual and textual features, semantically redundant text tokens are identified and pruned. Specifically, firstly, for visual features... and text features Perform a linear transformation and project it onto the query. s and keys Space, then calculate the scaled dot product similarity between the visual token and the text token:

[0058]

[0059] in, This represents the similarity score matrix. Further, the average similarity score between all visual tokens and text tokens is calculated to obtain the score vector.

[0060]

[0061] exist A higher score for a text token indicates that it receives significant attention across multiple visual tokens, meaning its semantic information may highly overlap with or be overly generic with visual features. Therefore, according to... Sort in descending order and remove the highest-scoring items. Text token, to obtain the final refined text features :

[0062]

[0063] in, Indicates based on the rating vector From the characteristics Remove the highest scorer from the list. Item, will Set it to 20.

[0064] The corresponding visual features are obtained at different layers using the DTR module. and refined text features In order to effectively integrate visual features and text features A visual language fusion network was designed, such as Figure 2 As shown. This network enhances intra-modal and inter-modal interactions through a multimodal fusion module (MFM) and aggregates multi-scale features in a top-down manner. Specifically: in MFM, such as Figure 5 As shown, refined text features of the same layer and visual features The features are then spliced ​​together to form preliminary multimodal features. Subsequently, a self-attention mechanism was used to fully extract relevant information within each modality, resulting in optimized multimodal features. Then, the enhanced visual features are extracted from them. Next, As a query Text features As a key Sum By leveraging the cross-attention mechanism to further integrate information between modalities, and fully utilizing the prior knowledge contained in the text features to enhance the localization of key discriminative regions in the visual features, this process can be represented as:

[0065]

[0066]

[0067]

[0068]

[0069] in, This indicates the extraction of the former from multimodal features. The visual portion of each location. The dimension representing the key is used to scale the attention score. Finally, a top-down architecture is adopted, using 3×3 convolutions to progressively downsample and fuse multi-scale features, transferring and aggregating prior knowledge from each level to the final visual features. .

[0070] Will After processing by the classification head, fine-grained image classification and prediction are performed.

[0071] Ultimately, the token set will be enhanced. Through a classification head Fine-grained image classification is performed.

[0072] Throughout the process, an end-to-end optimization strategy is adopted, and the overall loss is composed of classification loss. Combating losses and rating loss Together they form a whole. The classification loss uses cross-entropy supervised fine-grained classification.

[0072]

[0073] in, For real labels, To predict probabilities for the model, modality classifiers are introduced in different layers to effectively align cross-modal feature distributions. And based on adversarial loss, it reduces the distribution difference between image and text features:

[0074]

[0075] in Modal labels for images (0) or text (1). Additionally, the rating loss... Used to optimize rating tokens at different levels Discriminative ability:

[0076]

[0077] in, For the first Predicting the rating token in the layer classification header. Finally, the overall loss function is:

[0078]

[0079] in and The weights for the adversarial loss and the scoring loss are set to 0.2 and 0.5, respectively.

[0080] The ACC metric is used for evaluation on different datasets, and its specific definition is as follows:

[0081]

[0082] in, This represents the number of samples correctly classified by the model. This represents the total number of all test samples. A higher ACC value indicates higher accuracy in model predictions.

[0083] Table 1 shows the test results on the NABirds dataset: Table 1: Comparison of different methods on the NAAbirds dataset

[0084] Table 2 shows the test results for CUB-200-2011: Table 2: Comparison of different methods on the CUB-200-2011 dataset

[0085] Table 3 shows the test results at Stanford Cars: Table 3: Comparison of different methods on the Stanford Cars dataset

[0086] As shown in Tables 1, 2 and 3, the present invention demonstrates superior performance compared to other works in the same field in fine-grained image classification.

[0087] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A fine-grained image classification method based on large model enhancement, characterized in that: The method comprises the following steps: Step (1): constructing a visual description generation method; interact with MLLM in the form of question and answer, and generate multi-level visual description for the input image; Step (2): according to the multi-level visual description obtained in step (1), process the text features through the text encoder; input the image through the backbone model to extract multi-level image features; Step (3), according to the text features and image features obtained in step (2), construct a network model; the network model includes visual description generation, double-guided text feature refinement, and visual language fusion; Step (4), according to the network model architecture of step (3), construct the double-guided text feature refinement module, which includes three sub-modules: feature alignment module, task-guided text feature selector, and similarity-guided text feature pruner; Step (5), according to the network model architecture of step (3), construct the visual language fusion network, which includes several multi-modal fusion modules and convolutional layers; Step (6): input the data obtained in step (2) into the model in step (3) for training Step (7): use the model trained in step (6) for fine-grained image recognition, input the fine-grained image into the model, and output the prediction result to realize high-precision fine-grained image classification.

2. The method of claim 1, wherein the method is based on a large model enhancement. The visual description generation method of step (1) is as follows: Step (1-1), first, in order to facilitate the understanding and generation of multi-modal large model and the visual description conforming to the expert knowledge, three layers of key prompt words are constructed; Step (1-2), then, the key prompt words obtained in step (1-1) are embedded into the corresponding templates designed with care, and finally three layers of prompt sentences are formed; Step (1-3), for the input image, the prompt sentences obtained in step (1-2) are used to guide the MLLM to generate multi-level visual description containing expert discriminant rules through question and answer.

3. The large model enhancement-based fine-grained image classification method according to claim 1, wherein, The specific operation method of text and image processing in step (2) is as follows: Step (2-1), the multi-level text features containing expert knowledge are extracted from the multi-level visual description obtained in step (1-3) through the text encoder; Step (2-2), the multi-level visual features are extracted from the input image in step (1-3) using the visual backbone model.

4. The method of claim 1, wherein the method is based on a large model enhancement-based fine-grained image classification. The data processing flow of the double-guided text feature refinement module in step (4) is as follows: Step (4-1), using the feature alignment module, the text features and visual features obtained in step (2) are aligned through adversarial learning; Step (4-2), using the task-guided text feature selector, the semantic features highly related to the task in the text features obtained in step (4-1) are retained through the adaptive scoring mechanism; Step (4-3), the similarity-guided text feature pruner analyzes the similarity between the text features obtained in step (4-2) and the visual features in step (4-1) to remove the text features highly coinciding with the visual features, thereby avoiding allocating more weights to redundant information; Step (4-4), a corresponding dual-guided text feature refinement module is constructed for each level of visual features and text features, and steps (4-1) to (4-3) are used to process the text and visual features.

5. The method of claim 1, wherein, The data processing flow of the visual language fusion network in step (5) is as follows: Step (5-1), the network enhances the intra-modal and inter-modal interaction of the visual and text features obtained from step (4) through a multi-modal fusion module MFM to obtain multi-scale fusion features; Step (5-2), the multi-scale fusion features are aggregated in a top-down manner; The multi-modal fusion module in step (5-1) is: Step (5-1-1), the refined text features obtained in step (4-3-2) are spliced with the visual features in step (4-1) to form preliminary multi-modal features; Step (5-1-2), the multi-modal features obtained in step (5-1-1) are fully mined for intra-modal related information using a self-attention mechanism to obtain optimized multi-modal features, and then the enhanced visual feature part is extracted from them; Step (5-1-3), taking the strengthened visual features in step (5-1-2) as a query Step (4-3-2), taking the refined text features obtained in step (4-3-1) as a key and value , and further fusing the information between the modalities by using a cross-attention mechanism to obtain the features after text and visual fusion; The top-down manner in step (5-2) is: Step (5-2-1), the multi-scale features obtained from the multi-modal fusion modules at different levels in step (5-1) are progressively down-sampled and fused through multiple 3x3 convolutions to obtain the final visual features.

6. The method of claim 1, wherein the method is based on a large model enhancement-based fine-grained image classification. The model training process mentioned in step (6) is: Step (6-1) first inputs the fine-grained image into the model for inference; Step (6-2) inputs the results obtained in step (6-1) into the loss function for loss calculation; Step (6-3) inputs the loss obtained in step (6-2) into the AdamW optimizer for backpropagation to update the network parameters, and the learning rate is updated using the cosine annealing algorithm; Step (6-4) repeats the above steps (6-1) to (6-3), and tests every certain number of rounds to save the current optimal model parameters.

7. The method of claim 1, wherein the method is based on a large model enhancement-based fine-grained image classification. The fine-grained image classification process mentioned in step (7) is: Step (7-1) loads the optimal model parameters saved in step (6) into the network; Step (7-2) inputs the fine-grained image into the network obtained in step (7-1) to obtain the prediction result of the image category.

8. The method of claim 4, wherein the method further comprises: The feature alignment module in step (4-1) is: Step (4-1-1), the text features and visual features obtained in step (2) are respectively mapped to the same dimension representation through two independent multi-layer perceptrons (MLP); Step (4-1-2), an adversarial mechanism is constructed between the modal classifier and the MLP, and the MLP is trained to generate features that are difficult for the modal classifier to distinguish, so as to align the text features and visual features; The construction method and processing flow of the multi-layer perceptron in step (4-1-1) are: Step (4-1-1-1), each MLP consists of a linear layer and a layer, projecting both the textual features and visual features obtained in step (2) into a unified 512-dimensional space; The construction method and processing flow of the modal classifier in step (4-1-2) are: Step (4-1-2-1), the modal classifier is composed of two linear layers and an activation function, which first maps the text features and visual features obtained from step (4-1-1) from 512 dimensions to 256 dimensions, and then outputs a 2-dimensional vector for modal discrimination. Step (4-1-2-1), the modal classifier is composed of two linear layers and an activation function, which first maps the text features and visual features obtained from step (4-1-1) from 512 dimensions to 256 dimensions, and then outputs a 2-dimensional vector for modal discrimination.

9. The method of claim 4, wherein the method is based on a large model enhancement-based fine-grained image classification method. The task-guided text feature selector in step (4-2) is: Step (4-2-1), first introduce a learnable scoring token on the text feature aligned in step (4-1-2) to evaluate the importance of each text token; Step (4-2-2), concatenate the text features aligned in step (4-1-2) and the score tokens introduced in step (4-2-1), and map it to the query by a linear transformation , key and value form; Step (4-2-3) is the query obtained through step (4-2-2). ,key Calculate query-key dot product similarity and apply Normalization yields the attention weight matrix. Step (4-2-4), in the attention weight matrix in step (4-2-3), extract the score token The attention distribution vector of the text token, and retain according to its score Important text tokens to build refined features; Step (4-2-5), to ensure the validity of the scoring token and optimize its representation, the scoring token obtained in step (4-2-4) is multiplied by the attention distribution vector of the text token obtained in step (4-2-2) to aggregate the key information that the scoring token pays attention to, and then, after linear transformation processing, the original scoring token is added to the residual connection to obtain the representation of the updated scoring token ; Step (4-2-6), input the updated scoring token in step (4-2-5) into the classification head and supervise the training through cross-entropy loss.

10. The method of claim 4, wherein, The similarity-guided text feature pruner in step (4-3) is: Step (4-3-1), first, the text feature obtained in step (4-2-4) and the visual feature in step (4-1) are linearly transformed and projected to the query and key space, and then the scaled dot product similarity between the visual token and the text token is calculated to obtain a similarity score matrix; Step (4-3-2), using the similarity score matrix obtained in step (4-3-1), calculate the average similarity score of all visual token pairs to the text token to obtain a scoring vector; Step (4-3-3), order the text tokens in descending order according to the score vector of step (4-3-2), remove the top text tokens with the highest scores, and obtain the final refined text features.

Images