Small sample image classification method and system based on large model hierarchical semantic enhancement
This method, which uses a large-model hierarchical semantic enhancement for few-sample image classification, leverages a two-layer semantic construction and an RL-gated attention module to dynamically fuse visual and linguistic features. This addresses the shortcomings of visual feature extraction and cross-modal semantic fusion in few-sample image classification, achieving higher classification accuracy and generalization ability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANDONG UNIV
- Filing Date
- 2026-02-03
- Publication Date
- 2026-05-15
AI Technical Summary
Existing few-sample image classification methods have shortcomings in visual feature extraction and cross-modal semantic fusion, especially in scenarios with a single labeled sample where classification performance is significantly bottlenecked. Furthermore, existing cross-modal fusion schemes are difficult to adaptively perform visual-language alignment, resulting in limited semantic gain.
We adopt a hierarchical semantic enhancement method based on a large model. We generate fine-grained attribute and global description text through two-layer semantic construction, and use RL gated attention module to dynamically fuse visual and linguistic features to achieve hierarchical semantic alignment from low-level attributes to high-level description. Combined with reinforcement learning adaptive gating mechanism, we dynamically balance the weight and path of cross-modal fusion.
It significantly improves classification accuracy and generalization ability in small sample scenarios. By complementing fine-grained attributes and global descriptions, it solves the problems of single semantic level and insufficient fusion in existing methods, and achieves more accurate cross-modal alignment and classification decision.
Smart Images
Figure CN122049503A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and artificial intelligence technology, and particularly relates to a few-sample image classification method and system based on large-model hierarchical semantic enhancement. Background Technology
[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.
[0003] Few-Shot Learning (FSL) is an important research direction in the field of computer vision. Its core goal is to enable the model to accurately classify new categories of images that have never been seen before, relying only on a very small number of labeled support samples. It aims to break through the dependence of traditional supervised learning on large-scale labeled data and simulate the cognitive mode of humans to quickly learn new concepts based on limited experience.
[0004] Existing few-shot image classification methods mainly revolve around two main approaches: visual representation optimization and cross-modal semantic enhancement. Early methods focused on pure visual feature mining, with the core idea being to project both support and query samples into a shared embedding space and achieve classification by measuring the distance between samples. The key lies in extracting class-related discriminative visual features and reducing intra-class discriminative interference. However, limited by a small number of support samples, visual representations often struggle to capture sufficient class discriminative information, especially in 1-shot scenarios with only a single labeled sample, where classification performance faces a significant bottleneck.
[0005] To compensate for the shortcomings of purely visual methods, researchers have begun to explore the introduction of cross-modal semantic information, such as natural language, to enhance the generalization ability of models by fusing the complementarity of visual and linguistic modalities. One typical approach is to use fixed templates (such as "Aphoto of {CLASS}") to encode category names, generating semantic prototypes to supplement the information gaps in visual prototypes. Another advanced approach leverages Large Language Models (LLMs) to expand concise category names into textual descriptions containing high-level semantics, thereby improving the quality of semantic guidance. However, these approaches often neglect low-level discriminative patterns, which are crucial for extracting category-specific features. Another method attempts to generate fine-grained category entities to replace high-level semantics. However, these methods rely solely on low-level or high-level semantic embeddings, failing to progressively align visual features at different levels with corresponding semantic levels. More critically, existing cross-modal fusion schemes mostly employ static fusion mechanisms, making it difficult to perform adaptive visual-linguistic alignment during cross-layer processes, resulting in limited semantic gain. Summary of the Invention
[0006] To overcome the shortcomings of the existing technologies, this invention provides a few-sample image classification method and system based on hierarchical semantic enhancement of large models. It aims to realize hierarchical semantic construction from low-level attributes to high-level descriptions, and to dynamically fuse visual and linguistic features through reinforcement learning adaptive gating mechanism to improve classification accuracy and generalization ability in few-sample scenarios.
[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a few-sample image classification method based on hierarchical semantic enhancement of a large model; Few-sample image classification methods based on large-model hierarchical semantic enhancement include: Receive a support set and a query set. For each category in the support set, perform two-layer semantic construction to generate a fine-grained attribute set and a global description text corresponding to the category. Visual feature sequences of support set and query set images are extracted using a visual backbone network, and the fine-grained attribute set and global descriptive text are encoded using a text encoder to obtain the corresponding text semantic feature sequences. The visual feature sequence of the support set and the text semantic feature sequence are dynamically fused by the RL gated attention module to obtain the fused features; The fused features are input into the circular classifier of the visual backbone network. The similarity between the query samples in the query set and the prototypes of each category is calculated by the prototype classifier. The query samples are then classified based on the similarity.
[0008] As a further technical solution, for each category in the support set, a two-layer semantic construction is performed to generate a fine-grained attribute set and a global descriptive text corresponding to the support category, including: The category names and supporting samples in the support set are used as conditions to input into the large language model to generate fine-grained attributes that describe candidate features. A progressive Top-k selection strategy is adopted to iteratively select the most relevant attributes from the candidate fine-grained attributes, forming a set of fine-grained attributes after selection. The filtered set of fine-grained attributes is input into the large language model to obtain a coherent global category description text.
[0009] As a further technical solution, a progressive Top-k selection strategy is adopted to iteratively filter the most relevant attributes from the candidate fine-grained attributes, forming a set of filtered fine-grained attributes, including: Initialize the semantic template with preset category prompts; For each candidate attribute, it is encoded into a text vector along with the current semantic template, and the cosine similarity between the two is calculated as the relevance score of that attribute. Select the attribute with the highest relevance score, add it to the current semantic template, update the semantic template, and iterate until a preset number of attributes are selected to complete the filtering.
[0010] As a further technical solution, a fusion feature is obtained by dynamically fusing visual feature sequences and text semantic feature sequences through an RL-gated attention module, including: The input visual feature sequence and text semantic feature sequence are normalized. Two attention paths are computed in parallel. The first path is an image-guided path, which uses text features as queries and visual features as keys and values to perform cross-attention computation. The second path is a text-guided path, which uses text features as queries, keys, and values to perform self-attention computation. Based on the visual features, text features, and global similarity between the two of the current layer, gate coefficients are generated through a lightweight policy network. The outputs of the two attention paths are weighted and fused using the gating coefficients to obtain the fused feature sequence.
[0011] As a further technical solution, the gating coefficient generation strategy is trained using a reinforcement learning method, and its reward function is defined as:
[0012] in, For the reward function; It is the actual text embedding obtained by encoding with a text encoder; It is a linear projection used to map fused features after average pooling to the same text space; For average pooling; Features of fusion; and These represent the query accuracy when the current step is merged with the previous step, respectively. , This is used as a weight to balance alignment and boosting.
[0013] As a further technical solution, the fused features are input into a circular classifier of the visual backbone network. A prototype classifier calculates the similarity between the query samples in the query set and the prototypes of each category. Based on this similarity, the query samples are classified, including: The fused features output by the adaptive RL gated attention module are injected into the backbone network through residual connection and splicing operations to obtain enhanced visual features of the query samples. Based on the enhanced support set visual features, the prototype vector for each category is calculated; The prototype classifier is used to calculate the cosine similarity between the enhanced visual features of the query sample and the prototype vectors of each category. Based on the similarity, the query sample is classified into the category with the highest similarity.
[0014] A second aspect of the present invention provides a few-sample image classification system based on hierarchical semantic enhancement of a large model.
[0015] Few-sample image classification systems based on large-model hierarchical semantic enhancement include: The input module is configured to receive a support set and a query set, and for each category in the support set, perform two-layer semantic construction to generate a fine-grained attribute set and a global description text corresponding to the category. The two-layer semantic construction module is configured to: extract visual feature sequences from the support set and query set images using a visual backbone network, and encode the fine-grained attribute set and global descriptive text using a text encoder to obtain the corresponding text semantic feature sequences. The adaptive fusion module is configured to dynamically fuse the visual feature sequence of the support set and the text semantic feature sequence through the RL gated attention module to obtain fused features. The prototype calculation and classification module is configured to: input the fused features into the prototype classifier of the visual backbone network, calculate the similarity between the query samples in the query set and the prototypes of each category through the prototype classifier, and classify the query samples based on the similarity.
[0016] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of the few-sample image classification method based on large-model hierarchical semantic enhancement as described in the first aspect of the present invention.
[0017] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the small sample image classification method based on large model hierarchical semantic enhancement as described in the first aspect of the present invention.
[0018] The fifth aspect of the present invention provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the steps in the few-sample image classification method based on large-model hierarchical semantic enhancement described in the first aspect of the present invention.
[0019] The above one or more technical solutions have the following beneficial effects: (1) This invention constructs a two-layer semantic system of fine-grained attributes and global description through the DSC module, realizing complementary enhancement of semantic information. On the one hand, by leveraging the strong semantic generation capability of the large language model, it mines category-specific low-level discriminative attributes, providing accurate clues for fine-grained alignment of visual features; on the other hand, through a progressive Top-k selection strategy, it iteratively filters the most discriminative attributes based on cosine similarity, effectively eliminating redundant and irrelevant generated content, and suppressing the interference of semantic illusion on classification decisions from the source. The resulting structured low-level attributes and coherent high-level descriptions provide high-quality, multi-level semantic support for cross-modal alignment, solving the defect of single semantic level in existing methods.
[0020] (2) This invention uses an RLA module to model cross-modal fusion as a sequential decision-making process, and dynamically balances the weights and paths of visual-language fusion through an adaptive gating mechanism trained by reinforcement learning. In the shallow layers of the visual network, the gating strategy focuses on guiding the alignment of fine-grained attributes with local visual features, strengthening the capture of category-specific details; in the deep layers of the network, it shifts to matching global semantic descriptions with the overall visual structure, highlighting global differences between categories. This hierarchical and dynamic alignment method perfectly adapts to the hierarchical evolution characteristics of visual features from local details to global structure, significantly improving the utilization efficiency of cross-modal information compared to existing static fusion mechanisms, and maximizing semantic gain.
[0021] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0023] Figure 1 This is a flowchart of the method in the first embodiment.
[0024] Figure 2 This is a system structure diagram of the second embodiment. Detailed Implementation
[0025] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0026] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.
[0027] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.
[0028] The overall approach proposed in this invention is as follows: Through two-layer semantic construction (DSC), under the common constraints of category names and supporting samples, large language models (LLMs) are driven to generate discriminative attributes. The most relevant attributes are progressively selected and further synthesized into a coherent category description. This process provides complementary low-level attributes and high-level descriptions, thereby achieving both fine-grained alignment and overall category understanding. RL-gated attention (RLA) is used to dynamically integrate the two-layer semantics across different layers of the visual network, and cross-modal fusion is expressed as a sequential decision-making process. A lightweight strategy trained through contextual reinforcement learning adaptively adjusts the contributions of self-attention and cross-attention to fuse text and visual tokens. Thus, shallow layers refine local attributes, while deeper layers emphasize global semantics, achieving more accurate cross-modal alignment.
[0029] Example 1 This embodiment discloses a few-sample image classification method based on hierarchical semantic enhancement of a large model, so as to achieve hierarchical and dynamic cross-modal alignment of semantics from low to high levels.
[0030] like Figure 1 As shown, a few-sample image classification method based on large-model hierarchical semantic enhancement includes: Step S1: Receive the support set and query set. For each category in the support set, perform two-layer semantic construction to generate a fine-grained attribute set and a global description text corresponding to the category.
[0031] The system receives a support set and a query set corresponding to a few-shot task. The support set contains N non-overlapping categories, with each category corresponding to K labeled RGB images as support samples. The query set consists of unlabeled RGB images to be classified. All input images are uniformly scaled to 224×224 pixels to fit the input requirements of the visual backbone network; simultaneously, the standard name of each support category is extracted as the semantic input basis.
[0032] For each supported category, the Qwen2.5-VL-32B large language model is invoked, with input including prompts containing the category name and supporting samples, such as "What are the key discriminative attributes of this category in this image? Please list concise attribute points," thus obtaining a set of candidate attributes:
[0033] in, Represents a collection of attributes. Representing a large language model, This indicates a keyword used to indicate attribute discovery.
[0034] Then a progressive Top- Choose to retrieve fine-grained attributes. Specifically: initialize the semantic template. Each candidate attribute The input is fed into the CLIP text encoder to obtain a 512-dimensional attribute embedding; this embedding is then compared with the current template embedding. Cosine similarity score :
[0035] The template is updated by adding the attributes with the highest similarity. Repeat the iterative filtering process described above until the k most discriminative attributes are retained, forming the filtered attribute set:
[0036] in, This is the filtered set of fine-grained attributes; For a progressive Top-k selection strategy, this iterative enhancement ensures that only the most discriminative attributes are retained, while suppressing illusory or redundant attributes.
[0037] To further capture the overall semantics consistent with local details, based on the filtered set of fine-grained attributes, the Qwen2.5-VL-32B large language model is invoked again to process the selected set of fine-grained attributes. In summary, it can be described in a fluent and scientific manner. As shown below:
[0038] in, These are abstract prompts; For large models; These are cue words used for summarizing and generalizing; these global descriptions complement local attributes, thus generating a two-layer semantic hierarchy from low to high levels and jointly enhancing visual features.
[0039] Through the above process, a two-layer semantic information with both local detail recognition and global semantic integrity is output for each supported category. The fine-grained attribute set is used for low-level cross-modal alignment, and the global description text is used for high-level cross-modal alignment.
[0040] Step S2: Encode the fine-grained attribute set and the global description text using a text encoder to obtain the corresponding text semantic feature sequence.
[0041] For the filtered set of fine-grained attributes Standardize the format to create a list of attribute text; standardize the global category description text. Standardize punctuation to form a complete single paragraph of text.
[0042] The preprocessed list of attribute texts is input into the text encoder one by one, and a 512-dimensional feature vector is generated for each attribute text fragment to obtain the attribute semantic feature set; the preprocessed global description text is input into the text encoder to generate a single 512-dimensional global semantic feature vector.
[0043] Arrange the attribute semantic feature set according to the attribute filtering order, and then concatenate the global semantic features as the last element of the sequence to form a complete text semantic feature sequence.
[0044] Step S3: The visual feature sequence and the text semantic feature sequence are dynamically fused through the RL gated attention module to obtain the fused features.
[0045] During the extraction of visual features from the support set, the RL-gated attention mechanism dynamically fuses visual and textual features along the network depth. This adaptive mechanism gradually enables the model to focus on fine-grained local semantic details in the shallow layers of the network, while emphasizing global contextual semantics in the deeper layers, thereby achieving more accurate cross-modal fusion.
[0046] visual feature sequences and complete text semantic feature sequence A unified normalization process is performed to eliminate differences in statistical distribution between modalities and ensure consistent feature scales during the fusion process. First, a shared operator is used to normalize the two modalities to reconcile the statistics:
[0047]
[0048] in, These are the normalized visual features; These are the normalized semantic features of the text. This is a layer normalization operation.
[0049] Subsequently, image-guided paths and text-guided paths are constructed, both employing the standard scaled dot product attention mechanism:
[0050] in, For query vector; The key vector; It is a value vector; is the transpose; d represents the dimension.
[0051] In the image-guided path, textual semantic features are used as the query (Q), and visual features are used as the key (K) and value (V). Cross-attention is used to mine the correlation between textual semantics and image regions, outputting a visual representation. The calculation method is as follows:
[0052] in, For visual representation; , , It is a learnable linear projection matrix used to map features to a shared latent space.
[0053] In the text-guided path Derived from text tokens, a text-based refinement of semantic relationships is obtained. The calculation method is as follows:
[0054] in, For refined text representation; , This is the linear projection matrix corresponding to the text modality.
[0055] To adaptively balance the two cross-modal paths, RLA uses a random gating system to fuse their outputs. First, cross-modal compact states are computed by aggregating modal global information and aligned states using global average pooling (GAP) and cosine similarity, as shown below:
[0056] in, It is a cross-modal compact state; This indicates that the token sequence will be compressed into a global vector. This indicates the alignment between the computational visual and text global vectors. It is a lightweight MLP used to characterize the current cross-modal fusion context.
[0057] Based on compact state Hybrid weights are generated through a Beta policy network. The formula is:
[0058] in, For a Beta policy network, its mean is equal to ; The probability value output by the MLP. For the concentration parameter, the deterministic and exploratory nature of the weighting are controlled. Used for dynamic equilibrium and The percentage of contribution.
[0059] The dual-path representations are fused using dynamic weights to obtain the fused features. The formula is:
[0060] This fusion process enables layered alignment, and shallow network layers... Inclined towards larger values, with emphasis Enhance fine-grained attribute alignment, deep network Preferring smaller values, emphasizing This enhances global semantic alignment.
[0061] Fusion features Global average pooling is used to obtain the global semantic vector. Broadcast it to all visual tokens and sum the channel residuals, then add it to the output. The extended feature sequence is formed by concatenating the sequences, as shown in the formula:
[0062] Subsequently, the modulated and The sequences are then spliced together to form an extended sequence for use by subsequent Transformer layers.
[0063] The policy network parameters are optimized by designing a task-aware reward function, as shown below:
[0064] in, It is a 512-dimensional real text embedding obtained by CLIP text encoder. It is a linear projection used to fuse features after gaps. Mapped to the same text space. and These represent the query accuracy when merging the current result with the previous result. Weight Used for balance alignment and lifting.
[0065] The policy network is updated using the REINFORCE algorithm, and the policy gradient is:
[0066] in, It is an exponential moving average baseline used to reduce variance. For parameters The gradient operator; To optimize the objective; For expectations; The reward for time step t; This is the entropy regularization coefficient; the larger it is, the more random and exploratory the policy distribution is encouraged.
[0067] Regarding the training objective, the contribution of the RL loss to each block is aggregated and defined as:
[0068] in, It is the first The strategy for each block; The number of stacked blocks; For a query set in a single task; To determine the features of the q-th query sample at the th... The gated mixing coefficients and state vectors on each block. During training, It is applied in a hybrid soft-hard approach (sampling the stopping gradient path and averaging the differentiable path), while the expectation is used during the inference phase. Alternatives are used. Stacking multiple RLA blocks enables the policy to adaptively emphasize attribute-level cues at shallow levels and descriptive-level semantics at deeper levels, thereby achieving hierarchical visual-language alignment in few-shot learning.
[0069] Step S4: Input the fused features into the prototype classifier of the visual backbone network, calculate the similarity between the query samples in the query set and the prototypes of each category through the prototype classifier, and classify the query samples based on the similarity.
[0070] A prototype classifier is used as the training target. For each category... Its prototype embedding By analyzing the backbone network The average of the extracted support features is obtained as follows:
[0071] For a query sample The similarity between it and each prototype is measured using cosine similarity, and the similarity between them is measured using cosine similarity. Categorization The probability is defined as:
[0072] in, Let be the temperature parameter, and represent the cosine similarity function. The predicted label corresponds to the category with the highest probability. Finally, the total loss also includes the supervised loss: it measures the query sample... To be assigned to its true category probability The difference between the actual label and the corresponding real label is defined as:
[0073] in, This represents the trade-off hyperparameters for the RL weights. All learnable model parameters are minimized... To fine-tune the training, the episode is randomly sampled from the training categories.
[0074] Furthermore, in order to verify the effectiveness of the method of the present invention, This invention has been validated by a large number of experiments in three different few-sample image classification scenarios: (1) In the standard FSL scenario, four datasets were selected, including miniImageNet and tieredImageNet; (2) In the fine-grained FSL scenario, three datasets were selected, including CUB, Cars and Dogs; (3) In the cross-domain FSL scenario, three datasets were selected, including CUB, Places and ChestX.
[0075] During training, this invention employs a common two-stage few-shot learning strategy, including large-scale pre-training and meta-fine-tuning. Visformer-Tiny, based on ViT, is selected as the visual backbone network. The text encoder uses CLIP of ViT-B / 16 with an embedding dimension of 512. To provide additional semantic supervision, Qwen2.5-VL-32B is used to generate attribute-level and description-level text information. All input images are scaled to 224×224. Optimization is performed using AdamW with an initial learning rate of 5×10⁻⁶. 4. A cosine learning rate scheduler was used. Pre-training was performed for 300 epochs on tieredImageNet and 800 epochs on other datasets, with a batch size of 512 for both. Contextualized meta-fine-tuning was then performed for 100 epochs. In the RLA, the concentration parameter for the Beta distribution was set to κ=10, while λsim and λimp were set to 0.5 and 1.0, respectively. In the reinforcement learning gating, λ=0.1 was used to balance the RL objective, and τ=0.2 was set, both determined based on the validation set accuracy. All experiments were performed on an NVIDIA RTX 6000 Ada GPU.
[0076] This invention employs a contextual evaluation protocol widely used in few-shot learning. Specifically, 2000 classification tasks are uniformly sampled from new categories that do not overlap with the training categories. Each task follows a standard N-way K-shot setup, and 15 query samples are included for each category at evaluation time. The final performance is reported as the average classification accuracy across all sampled tasks, with a 95% confidence interval provided.
[0077] Table 1 shows the results on miniImageNet, tieredImageNet, and CIFAR-FS. Semantic-based methods (such as SP and SemFew) generally outperform purely metric learning or optimization-based methods, indicating the advantage of incorporating language priors into few-shot learning. DVLA-RL consistently achieves best or near-best accuracy across all datasets and in both 1-shot and 5-shot settings. For example, it achieves 81.69% / 88.25% on miniImageNet and 87.18% / 90.59% on CIFAR-FS, representing an improvement of 0.6%–2.8% compared to the strong baseline SemFew.
[0078] Table 1. Comparison results (%) of the method of this invention with other algorithms on different datasets.
[0079] These results demonstrate that the present invention can effectively supplement fine-grained attributes with global descriptions, while RL-gated attention can adaptively balance cross-modal fusion at different network depths. This hierarchical alignment enables the algorithm proposed in this example to suppress semantic illusions and better capture discriminative cues from limited samples, ultimately leading to stronger generalization ability in few-shot learning.
[0080] Example 2 This embodiment discloses a few-sample image classification system based on large-model hierarchical semantic enhancement; like Figure 2 As shown, a few-sample image classification system based on large-model hierarchical semantic enhancement includes: The input module is configured to receive a support set and a query set, and for each category in the support set, perform two-layer semantic construction to generate a fine-grained attribute set and a global description text corresponding to the category. The two-layer semantic construction module is configured to: extract visual feature sequences from the support set and query set images using a visual backbone network, and encode the fine-grained attribute set and global descriptive text using a text encoder to obtain the corresponding text semantic feature sequences. The adaptive fusion module is configured to dynamically fuse the visual feature sequence of the support set and the text semantic feature sequence through the RL gated attention module to obtain fused features. The prototype calculation and classification module is configured to: input the fused features into the prototype classifier of the visual backbone network, calculate the similarity between the query samples in the query set and the prototypes of each category through the prototype classifier, and classify the query samples based on the similarity.
[0081] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.
[0082] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps in the few-sample image classification method based on large-model hierarchical semantic enhancement as described in Example 1.
[0083] Example 4 The purpose of this embodiment is to provide an electronic device.
[0084] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the few-sample image classification method based on large model hierarchical semantic enhancement as described in Embodiment 1.
[0085] Example 5 Embodiment 5 of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the few-sample image classification method based on large model hierarchical semantic enhancement as described in Embodiment 1.
[0086] The steps and methods involved in the apparatuses of Embodiments 2, 3, 4, and 5 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.
[0087] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.
[0088] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.
Claims
1. A few-sample image classification method based on hierarchical semantic enhancement using a large model, characterized in that, include: Receive a support set and a query set. For each category in the support set, perform two-layer semantic construction to generate a fine-grained set of attributes and a global description text corresponding to the category. Visual feature sequences of support set and query set images are extracted using a visual backbone network, and the fine-grained attribute set and global descriptive text are encoded using a text encoder to obtain the corresponding text semantic feature sequences. The visual feature sequence and the text semantic feature sequence are dynamically fused through an RL-gated attention module to obtain fused features; The fused features are input into the prototype classifier of the visual backbone network. The prototype classifier calculates the similarity between the query samples in the query set and the prototypes of each category. The query samples are then classified based on the similarity.
2. The few-sample image classification method based on large-model hierarchical semantic enhancement as described in claim 1, characterized in that, For each category in the support set, a two-layer semantic construction is performed to generate a fine-grained set of attributes corresponding to the support category and a global descriptive text, including: The category names and supporting samples in the support set are used as conditions to input into the large language model to generate fine-grained attributes that describe candidate features. A progressive Top-k selection strategy is adopted to iteratively select the most relevant attributes from the candidate fine-grained attributes, forming a set of fine-grained attributes after selection. The filtered set of fine-grained attributes is input into the large language model to obtain a coherent global category description text.
3. The few-sample image classification method based on large-model hierarchical semantic enhancement as described in claim 2, characterized in that, A progressive Top-k selection strategy is adopted to iteratively filter the most relevant attributes from the candidate fine-grained attributes, forming a set of filtered fine-grained attributes, including: Initialize the semantic template with preset category prompts; For each candidate attribute, it is encoded into a text vector along with the current semantic template, and the cosine similarity between the two is calculated as the relevance score of that attribute. Select the attribute with the highest relevance score, add it to the current semantic template, update the semantic template, and iterate until a preset number of attributes are selected to complete the filtering.
4. The few-sample image classification method based on large-model hierarchical semantic enhancement as described in claim 1, characterized in that, The visual feature sequence and the text semantic feature sequence are dynamically fused through an RL-gated attention module to obtain fused features, including: The input visual feature sequence and text semantic feature sequence are normalized. Two attention paths are computed in parallel. The first path is an image-guided path, which uses text features as queries and visual features as keys and values to perform cross-attention computation. The second path is a text-guided path, which uses text features as queries, keys, and values to perform self-attention computation. Based on the visual features, text features, and global similarity between the two of the current layer, gate coefficients are generated through a lightweight policy network. The outputs of the two attention paths are weighted and fused using the gating coefficients to obtain the fused feature sequence.
5. The few-sample image classification method based on large-model hierarchical semantic enhancement as described in claim 4, characterized in that, The gating coefficient generation strategy is trained using reinforcement learning, and its reward function is defined as: in, For the reward function; It is the actual text embedding obtained by encoding with a text encoder; It is a linear projection used to map fused features after average pooling to the same text space; For average pooling; Features of fusion; and These represent the query accuracy rates when the current step is merged with the previous step, respectively. , This is used as a weight to balance alignment and boosting.
6. The few-sample image classification method based on large-model hierarchical semantic enhancement as described in claim 1, characterized in that, The fused features are input into the prototype classifier of the visual backbone network. The prototype classifier calculates the similarity between the query samples in the query set and the prototypes of each category. Based on the similarity, the query samples are classified, including: The fused features output by the adaptive RL gated attention module are injected into the backbone network through residual connection and splicing operations to obtain enhanced visual features of the query samples. Based on the enhanced support set visual features, the prototype vector for each category is calculated; The prototype classifier is used to calculate the cosine similarity between the enhanced visual features of the query sample and the prototype vectors of each category. Based on the similarity, the query sample is classified into the category with the highest similarity.
7. A few-sample image classification system based on hierarchical semantic enhancement using a large model, characterized in that, include: The input module is configured to receive a support set and a query set, and for each category in the support set, perform two-layer semantic construction to generate a fine-grained attribute set and a global description text corresponding to the category. The two-layer semantic construction module is configured to: extract visual feature sequences from support set and query set images using a visual backbone network, and encode the fine-grained attribute set and global description text using a text encoder to obtain the corresponding text semantic feature sequences. The adaptive fusion module is configured to dynamically fuse the visual feature sequence of the support set and the text semantic feature sequence through the RL gated attention module to obtain fused features. The prototype calculation and classification module is configured to: input the fused features into the prototype classifier of the visual backbone network, calculate the similarity between the query samples in the query set and the prototypes of each category through the prototype classifier, and classify the query samples based on the similarity.
8. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by a processor, the program implements the steps in the few-sample image classification method based on large model hierarchical semantic enhancement as described in any one of claims 1-6.
9. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the few-sample image classification method based on large model hierarchical semantic enhancement as described in any one of claims 1-6.
10. A computer program product comprising a computer program / instructions, characterized in that, When executed by a processor, the computer program / instruction implements the steps in the few-sample image classification method based on large model hierarchical semantic enhancement as described in any one of claims 1-6.