A small sample fine-grained image classification method based on multi-modal semantic screening and saliency fusion

Abstract

This invention discloses a small-sample fine-grained image classification method based on multimodal semantic filtering and saliency fusion, comprising the following steps: Step 1, using a multimodal semantic alignment mechanism to filter the most accurate text description from the original image to be enhanced, providing high-quality generation guidance conditions without semantic drift; Step 2: based on these guidance conditions, generating realistic enhanced images through saliency fusion to expand the training set, solving the problem of scarce small-sample data; Step 3: using a bilateral feature relation network for end-to-end feature learning and classification to achieve small-sample fine-grained image classification on the expanded training set. This invention features accurate semantic alignment, reasonable background logic, and natural foreground fusion.

Description

Technical Field

[0001] This invention belongs to the field of small-sample fine-grained image classification technology, specifically involving a small-sample fine-grained image classification method based on multimodal semantic screening and saliency fusion. Background Technology

[0002] Fine-grained visual classification (FGVC) is an important branch of computer vision. Its goal is to accurately distinguish subcategories belonging to the same base category, such as different breeds of birds, dogs, or car models. Unlike general image classification, fine-grained classification tasks rely heavily on subtle local visual features of objects (such as feather texture, beak color, or headlight shape).

[0003] The Few-Shot Dilemma and Traditional Data Augmentation: Training high-precision, fine-grained classification models typically requires massive amounts of well-labeled training data. In practical applications, acquiring and labeling image data in specialized fields is costly and difficult, leading to the "small-shot dilemma" that often results in overfitting and poor generalization ability. To address the data scarcity problem, data augmentation is a well-known solution. Traditional methods mainly include random cropping, rotation, flipping, and color dithering. While these methods are computationally inexpensive, they only perform geometric or photometric transformations in pixel space, failing to introduce new semantic information and having limited ability to expand the feature space, making them unsuitable for complex and ever-changing real-world environments.

[0004] In the prior art, the data augmentation method based on generative models (patent number: US12051487B2) is the closest technical solution to the current approach, which utilizes generative models to synthesize new samples to enrich training ensembles as generative adversarial network diffusion models develop.

[0005] Although this approach introduces new samples, it has significant drawbacks in fine-grained, small-sample tasks: (1) Semantic drift: Existing models often focus on global semantics while ignoring local details. For example, they cannot accurately preserve the subtle features of a specific species (such as the beak color or texture of a specific bird), resulting in the generated images being realistic but the category labels have changed ("image and text mismatch"), introducing serious label noise to the classifier.

[0006] (2) Lack of background logic and simplistic approach: Existing methods lack context awareness and often generate backgrounds that violate physical common sense (such as generating seabirds in a forest) or have overly simplistic backgrounds. Such background noise can interfere with the model's recognition of the foreground.

[0007] (3) Generating artifacts and unnatural fusion: In order to preserve foreground features, some methods adopt a simple "cut-paste" strategy or directly redraw, which easily produces obvious jagged edges and edge artifacts at the junction of foreground and background, destroying the natural statistical regularity of the image and affecting the training effect of the model.

[0008] In summary, the existing products have the following technical shortcomings: (1) Semantic drift and label noise problem: Since existing generative data augmentation methods usually directly use "category labels" to generate images, they lack depth alignment and verification of fine-grained visual features of the original image. As a result, although the generated image belongs to the category in a macroscopic sense, it loses key local details (such as the beak color and feather texture of a specific breed) in a microscopic sense, thus producing label noise that "does not match the text" and misleads the training of the classifier.

[0009] (2) Background logic deficiency and bias: Because existing methods lack the ability to reason about the context of an object's living environment, they simply generate or replace the background randomly, resulting in backgrounds that often violate physical common sense (such as generating water birds in a forest) or are too monotonous. This background noise not only fails to assist training, but also interferes with the model's recognition of the foreground, making it difficult for the model to decouple from the background dependency and limiting its generalization ability.

[0010] (3) Problems with generation artifacts and unnatural edges: Some existing methods use simple cut-and-paste or direct redraw strategies to preserve foreground features, lacking fine processing of the boundary between foreground and background, resulting in obvious jagged edges and artifacts in the synthesized image. These unnatural artificial traces disrupt the statistical regularity of the image, are easily captured by classification networks and used as a basis for classification, seriously affecting the model's performance on real data. Summary of the Invention

[0011] In order to overcome the shortcomings of the existing technology, the purpose of this invention is to provide a small sample fine-grained image classification method based on multimodal semantic screening and saliency fusion, which has the characteristics of accurate semantic alignment, reasonable background logic, and natural foreground fusion.

[0012] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A small-sample fine-grained image classification method based on multimodal semantic filtering and saliency fusion includes the following steps; Step 1: Using an image description model, the most accurate text description of the original image to be enhanced is selected based on a multimodal semantic alignment mechanism, providing high-quality generation guidance conditions without semantic drift. Step 2: Based on this guiding condition, realistic enhanced images are generated through saliency fusion to expand the training set and solve the problem of scarce small sample data; Step 3: Train the Bilateral Feature Relationship Network (BiFRN) using the training set, and then use the trained network for image classification.

[0013] In step 1, a fine-grained semantic alignment evaluation model is constructed, features are extracted, and the optimal fine-grained attribute description is selected from multiple candidate texts to ensure the semantic accuracy of the generated prompt.

[0014] Step 1 specifically involves: (1) Define variables: Let the original image to be enhanced be... Generated using image description models The set of candidate text descriptions is ; (2) Feature extraction: The pre-trained CLIP model is used to extract features from the original image and the candidate text respectively; (3) Based on the above unified shared feature space, a semantic alignment evaluation function is established to select the optimal fine-grained attribute description and ensure the semantic accuracy of the generated prompt.

[0015] The CLIP model employs a two-stream network architecture, including an image encoder dedicated to extracting visual features. A text encoder specifically designed for extracting semantic features The CLIP model maps and aligns visual and textual data into a unified, shared feature space.

[0016] Among them, let Represents the original image Visual feature vectors extracted by the image encoder Indicates the first Candidate text The semantic feature vectors extracted by the text encoder; based on the unified feature space mentioned above, a semantic alignment evaluation function is established, the optimization objective of which is to maximize the visual feature vectors of the original image. semantic feature vectors of candidate texts The cosine similarity between them is used to accurately select the optimal fine-grained attribute description. The calculation formula is as follows: in: : Represents the optimal fine-grained attribute description text obtained after rigorous calculation and filtering by the evaluation function; : Indicates generated by the image description model The set of candidate text descriptions is defined as follows: .

[0017] : Represents a set of candidate texts The first in Description text; : Represents the original image via image encoder After extraction, the visual feature vectors are mapped to the shared feature space; : indicates the first Candidate text Text encoder After extraction, the semantic feature vectors are mapped to the shared feature space; The cosine similarity is calculated by combining the dot product (inner product) of two feature vectors in a unified feature space with the norm denominator. The higher the similarity score, the more accurately the extracted text describes the subtle local features of the original image to be enhanced (such as the texture and color of a specific variety), thus obtaining the optimal text description and effectively avoiding semantic drift during the generation stage; the text description with the highest similarity score is extracted as the optimal text description.

[0018] Step 2 involves constructing a generative synthesis model: This involves generating realistic enhanced images through saliency fusion, increasing the number of samples in the support set, thereby expanding the sample size and improving the model's generalization ability. Specifically, Step 2 includes: Based on the optimal text description obtained in step 1, a clean foreground image is extracted from the original image using the Grounding-SAM collaborative framework. Combined with large language model logical inference and CLIP semantic filtering, an optimal background image consistent with fine-grained category semantics is obtained. Simultaneously, SAM generates a high-precision binary saliency mask for the corresponding foreground. Subsequently, two-dimensional affine operations such as angle transformation are performed on the clean foreground image to generate foreground variants with diverse poses. The final enhanced image... Defined as the weighted sum (Hadamard product) of the clean foreground image and the optimal background image. The specific calculation formula is as follows: in A clean foreground image extracted from the original image using the Grounding-SAM collaborative framework; The optimal background image obtained through LLM logic reasoning and CLIP filtering; : A high-precision binary saliency mask generated by SAM, where 1 represents the foreground region; : Foreground geometric transformation operations (such as rotation and scaling) introduced to increase the diversity of sample poses.

[0019] The geometric transformation operation is specifically implemented through a two-dimensional affine transformation, for any pixel coordinates in the clean foreground image. Transformed coordinates The calculation formula is as follows: in: This represents Hadamard, which is element-wise multiplication. This ensures that the foreground image is transformed after geometric transformation. Afterwards, its texture details were completely preserved; This method uses the reverse area of ​​the mask to fill the background, ensuring that background pixels do not cover the foreground, thus eliminating the jagged edges caused by traditional "cut and paste".

[0020] Step 3 involves constructing a small-sample classification metric model and training a bilateral feature relation network using the high-quality augmented data generated in the first two steps to complete the classification. These three steps form a complete closed loop: semantic filtering to ensure quality, synthetic augmentation to increase quantity, and metric learning to complete classification. Under conditions of extremely scarce labeled samples and complex backgrounds, robust fine-grained discriminative features are extracted to eliminate noise interference from the synthetic background, achieving high-precision small-sample fine-grained image classification.

[0021] Step 3 specifically involves: (1) Feature pair construction and nonlinear measurement: The image enhanced in step 2 Input a bilateral feature extraction network, and let the features of the query set samples be... Support set sample features are The two are concatenated along the channel dimension and then input into a relational module composed of a deep neural network. In this process, the sample relationship scores predicted by the network are obtained. This score characterizes the degree of matching between the query sample and the supporting samples in the fine-grained feature space.

[0022] Loss function design and optimization: The training objective of the model is to minimize the mean squared error (MSE) between the predicted relation scores and the true labels. Its loss function... The definition is as follows: The sample similarity score predicted by the network typically ranges from [0, 1]. : refers to the indicator function, when querying samples With supporting samples The value is 1 if they belong to the same category, and 0 otherwise. By optimizing the loss function, the model can learn fine-grained discriminative features that remain robust in complex synthetic contexts.

[0023] A small-sample fine-grained image classification system based on multimodal semantic screening and saliency fusion innovatively abandons the rigid "random transformation-direct training" pipeline in traditional data augmentation, instead constructing a deep closed-loop system containing three core modules. Specifically, the system mainly includes a fine-grained semantic alignment evaluation module, a generative saliency fusion enhancement module, and a small-sample classification metric module. First, the fine-grained semantic alignment evaluation module plays a role in semantic protection. It uses the CLIP model to extract the visual features of the original image and the semantic features of the candidate text and calculate the cosine similarity, thereby accurately selecting the optimal description containing micro-details such as "beak color" and "feather texture". This cuts off the semantic drift phenomenon that is very easy to occur when directly generating images using category labels in the traditional way. Secondly, the generative saliency fusion enhancement module, based on the above optimal description, combines the physically consistent background inferred from the large language model with the high-precision binary foreground mask extracted by Grounding-SAM, strictly adhering to the formula. The pixel-level seamless fusion completely eliminates the jagged edges and artifacts caused by the traditional "cut and paste" method; Finally, the few-shot classification metric module receives the high-quality, noise-free enhanced image output by the above modules. The query set and support set features are concatenated in the channel dimension using a Bilateral Feature Extraction Network (BiFRN), and the final classification is completed through end-to-end metric learning.

[0024] In terms of their interrelationship, these three modules are progressive and interdependent: the semantic alignment evaluation module provides the saliency fusion enhancement module with accurate and unbiased semantic guidance conditions, while the fusion enhancement module provides the small sample classification measurement module with a large number of high-quality training samples with rigorous logic. The three work together to significantly overcome the problem of feature extraction and classification under the condition of extremely scarce labeled samples.

[0025] The beneficial effects of this invention are: This invention utilizes the chain-like thinking ability of large language models to perform logical reasoning, accurately derive and generate contextual environments that conform to real physical laws, effectively expand the richness of the background and decouple the model from the background.

[0026] This invention utilizes the Grounding-SAM collaborative framework to extract high-precision binary saliency masks and performs weighted calculations strictly according to the reverse region of the mask. This achieves pixel-level seamless fusion of the foreground and the logical reasoning background, generating high-quality enhanced samples without artifacts and preventing overfitting of the classification network to unnatural edges. It overcomes the jagged edges and manual splicing marks caused by the traditional "cut-and-paste" strategy.

[0027] By combining the above technical means, this invention applies the generated high-quality, noise-free data to a bilateral feature extraction network for end-to-end learning. In 1-Shot and 5-Shot tasks on standard fine-grained datasets such as CUB-200-2011 and Stanford Dogs, it comprehensively outperforms benchmarks and traditional data augmentation schemes (for example, the accuracy is improved to 80.48% under the extremely challenging CUB 1-Shot setting), significantly improving the model's feature extraction capability and generalization performance under data-scarce conditions. Attached Figure Description

[0028] Figure 1 This is a diagram illustrating the overall framework of the enhanced network in this invention.

[0029] Figure 2 This is a combination of foreground and background diagrams for the present invention.

[0030] Figure 3 This is the foreground object image extracted in this invention.

[0031] Figure 4 This is a diagram of the text prompt filtering structure of the present invention. Detailed Implementation

[0032] The present invention will now be described in further detail with reference to the accompanying drawings.

[0033] like Figure 1 As shown, a small-sample fine-grained image classification method based on multimodal semantic screening and saliency fusion includes the following steps; Step 1: Feature extraction to construct a fine-grained semantic alignment evaluation model: This step aims to select the optimal description from multiple candidate texts to ensure the semantic accuracy of the generated guide words.

[0034] (1) Define variables: Let the original image to be enhanced be... Generated using image description models The set of candidate text descriptions is .

[0035] (2) Feature extraction: Features are extracted using the pre-trained CLIP model: : Represents the visual feature vector of the original image; : Represents the semantic feature vector of the k-th candidate text.

[0036] (3) Optimization Objective: Establish a semantic alignment evaluation function with the objective of maximizing the cosine similarity between the image and text in the feature space. Optimal fine-grained attribute description. The calculation formula is as follows: ； Other multimodal pre-trained models such as BLIP (Bootstrapping Language-Image Pre-training), ALIGN (ALarge-scale Image and Noisy-text Embedding), or ALBEF can be used to replace CLIP for extracting feature vectors and calculating similarity scores. Any method that utilizes a visual-language multimodal alignment mechanism for text filtering falls within the scope of this invention.

[0037] A visual question-answering model is used to extract fine-grained features through preset question templates and combine the answers into descriptive text.

[0038] Step 2, construct the generative synthesis model: Define variables: A clean foreground image extracted from the original image using the Grounding-SAM collaborative framework; The optimal background image obtained through LLM logic reasoning and CLIP filtering; : A high-precision binary saliency mask generated by SAM, where 1 represents the foreground region; : Foreground geometric transformation operations (such as rotation and scaling) introduced to increase pose diversity.

[0039] The final enhanced image Defined as the weighted sum of foreground and background (Hadamard product), the calculation formula is as follows: This represents Hadama, which is the product of corresponding elements.

[0040] This ensures that foreground objects, after geometric transformations, Afterwards, its texture details were completely preserved.

[0041] This method uses the reverse area of ​​the mask to fill the background, ensuring that background pixels do not cover the foreground, thus eliminating the jagged edges caused by traditional "cut and paste".

[0042] The original solution used a large language model for chain-like reasoning. A knowledge graph can be used to directly retrieve logically consistent background keywords by querying the "LocationOf" or "Habitat" relationship edges of category labels, thus replacing the LLM reasoning process.

[0043] The original solution uses a text-to-image model to generate a synthetic background. Real background images can be retrieved from search engines or image databases using the generated background keywords. Using the retrieved real background images to replace the generated background image, combined with a CLIP filtering mechanism, also falls within the scope of this invention.

[0044] Step 3: Construct a small sample classification metric model: This step utilizes a Bilateral Feature Relationship Network (BiFRN) for end-to-end feature learning and classification.

[0045] Feature pair construction: Constructing sample features from the query set Features of support set samples Concatenate along the channel dimension and input the relationship module. Obtain the predicted relation score .

[0046] Loss function: The training objective of the model is to minimize the mean squared error (MSE) between the predicted relation scores and the true labels. Loss function, loss function as follows: The sample similarity score predicted by the network typically ranges from [0, 1]. Indicator function, when querying samples With supporting samples The value is 1 if they belong to the same category, and 0 otherwise.

[0047] By optimizing the loss function, the model can learn fine-grained discriminative features that remain robust in complex synthetic contexts.

[0048] like Figure 3 The diagram illustrates the specific process of high-precision foreground object mask extraction in this invention. After inputting the original labeled image, the system first uses the Grounding DINO model to accurately locate and extract the bounding box of the target. Then, the located region information is input into the SAM model, which generates a high-precision binary saliency mask for the target.

[0049] In fine-grained classification tasks (such as distinguishing bird feathers or vehicle parts), the accuracy of the foreground contour directly determines the quality of subsequent synthesized data. This invention combines the Grounding-SAM collaborative framework, which not only achieves the extraction of a clean foreground from a complex background, but also generates a mask that far surpasses traditional saliency detection algorithms in edge fit, laying the foundation for subsequent high-quality, semantically drift-free image synthesis.

[0050] Figure 1 Comparative test results like Figure 1 Experimental results show that the proposed method outperforms existing benchmark methods (None), traditional data augmentation methods (such as AugMix), and direct text-to-image generation methods (Txt2img) in classification accuracy across the 1-Shot and 5-Shot tasks of three standard fine-grained datasets: CUB-200-2011, Stanford Dogs, and Stanford Cars. Particularly noteworthy is the extremely challenging 1-Shot setting, where existing Txt2img methods suffer from performance degradation due to semantic drift (e.g., falling below the benchmark on the CUB dataset). This invention, by introducing multimodal semantic filtering and logical background reasoning, not only addresses this deficiency but also achieves optimal performance across all datasets (e.g., improving to 80.48% on CUB 1-Shot). This demonstrates that the augmented samples generated by this framework possess higher semantic consistency and background richness, significantly enhancing the model's generalization ability and robustness under data-scarce conditions.

[0051] This invention proposes a fine-grained semantic attribute alignment method based on a "generation-filtering" closed loop (addressing the semantic drift problem): This invention does not merely generate text, but introduces the CLIP multimodal scoring mechanism to filter the generated text and select the most matching text.

[0052] This invention protects a technical solution that utilizes an image description model to generate candidate text and filters for the "optimal fine-grained attribute description" by calculating the cosine similarity between the feature vectors of the CLIP image encoder and the text encoder. This differs from traditional methods that directly generate images using category labels, ensuring that the prompt includes fine-grained features such as "beak color" and "feather texture." This invention leverages the reasoning capabilities of a large model to resolve background inconsistencies, rather than random generation. It also protects a technical process that utilizes a large language model combined with chain-like thinking prompts to infer habitat or environmental descriptions that conform to physical common sense based on foreground category labels. Furthermore, it protects a mechanism that uses CLIP to perform secondary filtering of the generated background image, ensuring semantic consistency of the background.

[0053] This invention combines the high-precision segmentation of Grounding-SAM with Img2Img light and shadow generation to achieve precise fusion in a mathematical sense.

[0054] This invention protects a mask-based image synthesis formula. High-precision binary masks are extracted using a foreground extraction module. The foreground after geometric transformation is obtained by using the Hadamard product. With optimal background A technical solution for pixel-level seamless blending. This point aims to protect the specific implementation methods for eliminating edge jaggedness and manual splicing traces.

[0055] A cascaded small-sample fine-grained image classification enhancement framework was constructed: The above modules are integrated and applied to a bilateral feature relationship network.

[0056] This invention protects a complete framework for applying the augmented data generated by the aforementioned semantic filtering, contextual reasoning, and saliency fusion to small-sample, fine-grained classification tasks. Specifically, it utilizes a bilateral feature relation network instead of a traditional prototype network, learning a classification architecture with non-linear similarity relationships by concatenating support set and query set features along the channel dimension.

[0057] As shown in Figure 2, this process takes the extracted foreground mask image as input and first feeds it into the foreground image transformation module. Through geometric operations such as angle transformation, various foreground variants with different poses and viewpoints are generated. Then, these transformed foregrounds are fused with different real background images at the pixel level. The mask is used to achieve seamless stitching of the foreground and background. Finally, multiple sets of enhanced images with diverse poses, rich backgrounds and no artifacts are synthesized. This expands the training data for small sample fine-grained image classification tasks, while avoiding the edge inconsistencies and semantic drift problems caused by traditional cut-and-paste methods.

[0058] As shown in Figure 4, the process first takes a fine-grained image to be enhanced (a red-winged blackbird in the image) as input, and generates five candidate text descriptions for matching the image through a generative large language model. At the same time, a baseline semantic text "a photo of the tag" is constructed based on the image's real labels. Then, using an image encoder and a text encoder, the original image, each candidate description, and the baseline text are mapped to a unified multimodal feature space. The cosine similarity cos1 between the image features and the candidate description features, and the cosine similarity cos2 between the image features and the baseline text features are calculated, and the average of the two is used as the final matching score of the candidate description. Finally, by comparing the scores of the five candidate descriptions, the description with the highest score is selected as the optimal guiding text. This text is combined with the baseline text to form the final generation guiding condition, thereby achieving accurate alignment and selection of fine-grained semantics of the image. This provides reliable semantic constraints for the subsequent generation of high-quality enhanced images and avoids semantic drift problems in the generation stage from the source.

Claims

1. A small-sample fine-grained image classification method based on multimodal semantic filtering and saliency fusion, characterized in that, Includes the following steps; Step 1: Using an image description model, the most accurate text description of the original image to be enhanced is selected based on a multimodal semantic alignment mechanism, providing high-quality generation guidance conditions without semantic drift. Step 2: Based on the aforementioned guiding conditions, generate realistic enhanced images through saliency fusion to expand the training set; Step 3: Train the Bilateral Feature Relationship Network (BiFRN) using the training set, and then use the trained network for image classification.

2. The small-sample fine-grained image classification method based on multimodal semantic screening and saliency fusion according to claim 1, characterized in that, Step 1 specifically involves: (1): Let the original image to be enhanced be... Generated using image description models The set of candidate text descriptions is ; (2): Use the pre-trained CLIP model to extract features from the original image and candidate text description respectively; map and align the extracted features into a unified shared feature space; (3) Based on the above unified shared feature space, a semantic alignment evaluation function is established to select the optimal fine-grained attribute description and ensure the semantic accuracy of the generated guide words.

3. The small-sample fine-grained image classification method based on multimodal semantic screening and saliency fusion according to claim 2, characterized in that, In step (2), the CLIP model adopts a two-stream network architecture, which includes an image encoder dedicated to extracting visual features. A text encoder specifically designed for extracting semantic features The CLIP model maps and aligns visual and textual features into a unified, shared feature space. Among them, let Represents the original image Visual feature vectors extracted by the image encoder Indicates the first Candidate text Semantic feature vectors extracted by the text encoder.

4. The small-sample fine-grained image classification method based on multimodal semantic screening and saliency fusion according to claim 3, characterized in that, In step (3), a semantic alignment evaluation function is established, the optimization objective of which is to maximize the visual feature vector of the original image. semantic feature vectors of candidate texts Cosine similarity between them is used to accurately select the optimal fine-grained attribute description. The calculation formula is as follows: in: : Represents the optimal fine-grained attribute description text obtained after rigorous calculation and filtering by the evaluation function; : Indicates generated by the image description model The set of candidate text descriptions is defined as follows: . : Represents a set of candidate texts The first in Description text; : Represents the original image via image encoder After extraction, the visual feature vectors are mapped to the shared feature space; : indicates the first Candidate text Text encoder After extraction, the semantic feature vectors are mapped to the shared feature space; The cosine similarity is calculated by combining the dot product of two feature vectors in a unified feature space with the norm denominator. The higher the similarity score, the more accurately the extracted text describes the subtle local features of the original image to be enhanced. The text with the highest similarity score is selected as the optimal text description.

5. The small-sample fine-grained image classification method based on multimodal semantic screening and saliency fusion according to claim 4, characterized in that, Step 2 specifically involves: Based on the optimal text description obtained in step 1, the clean foreground image of the original image is extracted through the Grounding-SAM collaborative framework. Combined with the logical reasoning of the large language model and CLIP semantic filtering, the optimal background image consistent with the fine-grained category semantics is obtained. At the same time, SAM generates a high-precision binary saliency mask for the corresponding foreground. A two-dimensional affine operation is then performed on the clean foreground image to generate foreground variants with diverse poses; the final enhanced image Defined as the weighted sum of the clean foreground image and the optimal background image, the calculation formula is as follows: in A clean foreground image extracted from the original image using the Grounding-SAM collaborative framework; The optimal background image obtained through LLM logic reasoning and CLIP filtering; : A high-precision binary saliency mask generated by SAM, where 1 represents the foreground region; : Foreground geometric transformation operations introduced to increase the diversity of sample poses.

6. The small-sample fine-grained image classification method based on multimodal semantic screening and saliency fusion according to claim 5, characterized in that, The geometric transformation operation is specifically implemented through a two-dimensional affine transformation, for any pixel coordinates in the clean foreground image. Transformed coordinates The calculation formula is as follows: in: This represents Hadamard, which is element-wise multiplication. This ensures that the foreground image is transformed after geometric transformation. Afterwards, its texture details were completely preserved; This method uses the reverse region of the mask to fill the background, ensuring that background pixels do not cover the foreground.

7. A small-sample fine-grained image classification method based on multimodal semantic screening and saliency fusion according to claim 6, characterized in that, Step 3 specifically involves: The image after step 2 enhancement Input a bilateral feature extraction network; Let the features of the query set samples be Support set sample features are The two are concatenated along the channel dimension and then input into a relational module composed of a deep neural network. In this process, the sample relationship scores predicted by the network are obtained. ; It represents the degree of matching between the query sample and the supporting samples in the fine-grained feature space.

8. A small-sample fine-grained image classification method based on multimodal semantic screening and saliency fusion according to claim 7, characterized in that, The training objective of the model is to minimize the mean squared error (MSE) between the predicted relation scores and the true labels; its loss function is... The definition is as follows: The sample similarity score predicted by the network typically ranges from [0, 1]. : refers to the indicator function, when querying samples With supporting samples The value is 1 if they belong to the same category, and 0 otherwise.

9. A small-sample fine-grained image classification method based on multimodal semantic screening and saliency fusion according to any one of claims 1-8, implemented by a small-sample fine-grained image classification system based on multimodal semantic screening and saliency fusion, characterized in that, It includes a fine-grained semantic alignment evaluation module, a generative saliency fusion enhancement module, and a small-sample classification metric module; The fine-grained semantic alignment evaluation module uses the CLIP model to extract the visual features of the original image and the semantic features of the candidate text, and calculates the cosine similarity to select the optimal description of the micro-details. The generative saliency fusion enhancement module, based on the above optimal description, combines the physically consistent background inferred from the large language model with the high-precision binary foreground mask extracted by Grounding-SAM, relying on the formula... Seamless pixel-level blending eliminates jagged edges and artifacts; The small sample classification metric module receives the high-quality, noise-free enhanced images output by the above modules. The query set and support set features are concatenated in the channel dimension using a bilateral feature extraction network, and the final classification is completed through end-to-end metric learning.

Images