Small sample image classification method and system based on multi-scale cross-modal cue enhancement

Abstract

The present invention discloses a small-sample image classification method and system based on multi-scale cross-modal cue enhancement, which relates to the technical field of small-sample image classification. The method comprises the following steps: obtaining a query image and a small-sample dataset of support images with labeled categories; inputting the obtained data into a trained small-sample image classification model; extracting the visual features of the query image and the enhanced visual features of each support image through a feature extractor in the model based on hierarchical multimodal cue enhancement; performing global average pooling on all the extracted enhanced visual features to obtain an enhanced visual feature prototype for each category; calculating a category similarity score based on the query image visual features and the enhanced visual feature prototype for each category; determining the category with the highest score as the classification result for the query image and outputting it; wherein the small-sample image classification model is trained using a global-local self-guided supervision mechanism. The present invention can significantly improve the accuracy and robustness of small-sample image classification.

Description

Technical Field

[0001] The present invention relates to the technical field of small sample image classification, and in particular to a small sample image classification method and system based on multi-scale cross-modal cue enhancement. Background Art

[0002] Deep learning has achieved tremendous success in computer vision fields, such as image classification. However, traditional deep learning-based image classification methods often rely heavily on large labeled datasets to achieve high classification accuracy. These datasets and their annotation costs are high and sometimes even impractical. For example, in rare disease diagnosis and industrial anomaly detection, the image data volume is extremely large, and labeling each image individually is extremely difficult. In contrast, humans can recognize new objects from just a few images. To mimic human learning ability and reduce reliance on annotated data, existing research has proposed few-shot learning (FSL) methods that learn from a limited amount of data. This method achieves accurate classification of unseen images by learning from a small number of labeled examples and generalizing to unseen examples. While these methods have achieved significant progress, achieving human-level performance remains challenging, especially in the one-shot scenario (i.e., single-shot learning) with only a single labeled example. Therefore, further optimization of few-shot learning methods is needed for few-shot image classification to improve overall classification performance.

[0003] To improve classification performance, existing few-shot learning methods have proposed using Transformer layers as classifiers. However, these classification models often struggle to generate robust and general visual representations with limited support data. To this end, other approaches have proposed using text modalities as auxiliary information to enhance visual features through adaptive mechanisms, such as fusing text and visual prototypes or learning classifiers using graph convolutional networks. However, due to the inherent distribution differences between modalities and the lack of deep cross-modal interaction in underlying features, these approaches struggle to capture complex cross-modal semantic relationships.

[0004] At the same time, this text-based approach often focuses solely on global supervision, neglecting local semantic modeling and fine-grained discriminative information. Modeling semantic information in local features helps avoid learning bias and non-generalizable features, significantly improving the model's generalization ability, which is crucial for small-sample learning. While existing vision-based methods also attempt to guide local feature learning through local supervision, they often face semantic confusion due to the lack of class-specific semantic information, making it difficult to improve supervision at specific locations.

[0005] In addition, existing small-sample learning methods lack the ability to process visual features at multiple scales, resulting in poor performance when dealing with different spatial changes. This lack of a hierarchical structure makes it difficult for the classification model to remain robust in the face of scale and position changes, which will also lead to a decrease in overall classification performance. Summary of the Invention

[0006] To address the deficiencies of the above-mentioned prior art, the present invention provides a small-sample image classification method and system based on multi-scale cross-modal cue enhancement, and proposes a hierarchical multimodal cue (HMP) and a global-local self-guided supervision (GLA) mechanism. These methods make full use of cross-modal information and conduct deep cross-modal interaction through the hierarchical multimodal cue mechanism to capture complex cross-modal semantic relationships, which are used to enhance visual features. The hierarchical structure is used to further improve the robustness to spatial changes. At the same time, the global-local self-guided supervision mechanism is used to select global and local discriminative features to generate universal and generalized visual representations. The above methods significantly improve the accuracy and robustness of small-sample image classification.

[0007] In a first aspect, the present invention provides a small sample image classification method based on multi-scale cross-modal cue enhancement.

[0008] A small sample image classification method based on multi-scale cross-modal cue enhancement, comprising:

[0009] Obtain a query image and a small sample dataset of support images with labeled categories;

[0010] The query image and the small sample dataset of support images are input into the trained small sample image classification model. The visual features of the query image and the enhanced visual features of each support image are extracted by the feature extractor ViT in the model based on hierarchical multimodal cue enhancement. All the extracted enhanced visual features are globally averaged and pooled to obtain the enhanced visual feature prototype of each category. The category similarity score is calculated based on the visual features of the query image and the enhanced visual feature prototype of each category. The category with the highest score is determined and output as the classification result of the query image.

[0011] The small sample image classification model is trained using a global-local self-guided supervision mechanism.

[0012] In a second aspect, the present invention provides a small sample image classification system based on multi-scale cross-modal cue enhancement.

[0013] A small-sample image classification system based on multi-scale cross-modal cue enhancement, comprising:

[0014] A data acquisition module is used to obtain query images and a small sample dataset of support images with annotated categories;

[0015] An image classification module is configured to input a query image and a small sample dataset of support images into a trained small sample image classification model. The model's feature extractor, ViT, based on hierarchical multimodal cue enhancement, extracts the visual features of the query image and the enhanced visual features of each support image. All extracted enhanced visual features are globally averaged and pooled to obtain an enhanced visual feature prototype for each category. A category similarity score is calculated based on the query image visual features and the enhanced visual feature prototypes of each category. The category with the highest score is determined and output as the classification result for the query image.

[0016] The small sample image classification model is trained using a global-local self-guided supervision mechanism.

[0017] In a third aspect, the present invention further provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps of the method described in the first aspect are completed.

[0018] In a fourth aspect, the present invention further provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the method described in the first aspect.

[0019] One or more of the above technical solutions have the following beneficial effects:

[0020] 1. The present invention provides a small sample image classification method and system based on multi-scale cross-modal cue enhancement, proposes a hierarchical multimodal cue HMP and a global-local self-guided supervision GLA mechanism, fully utilizes cross-modal information, and conducts deep cross-modal interaction through the hierarchical multimodal cue mechanism to capture complex cross-modal semantic relationships, which are used to enhance visual features, and further improve the robustness to spatial changes through a hierarchical structure. At the same time, through the global-local self-guided supervision mechanism, global and local discriminative features are selected, and thereby accelerated by semantic information-guided label dependency learning to generate general and generalized visual representations. The above method significantly improves the accuracy and robustness of small sample image classification, and solves the problems of poor classification performance caused by insufficient cross-modal semantic interaction, global and local supervision limitations, and insufficient spatial robustness in the existing technology.

[0021] 2. The present invention adopts a hierarchical multimodal prompt (HMP) mechanism. In each layer of the multimodal prompt MP, a pre-trained visual-language CLIP model is used to encode category text prompts to extract rich semantic information from the category name. Then, a learnable visual prompt is introduced at each scale of the extracted multi-scale visual features to adaptively adjust the text prompts during the training process to alleviate the distribution differences of cross-modal prompts. At the same time, the multimodal prompt MP allows the model to align the text prompts with the visual prompts when fusing the text prompts and visual prompts, thereby generating unbiased multimodal prompts and promoting the interaction between cross-scale multimodal prompts and visual features. Through the multi-layer multimodal prompt MP structure, a hierarchical multimodal prompt (HMP) module is constructed to improve spatial robustness.

[0022] 3. The present invention introduces a hierarchical multimodal cueing mechanism into the feature extractor / model (Vision Transformer, ViT), which enables the feature extractor to focus on class-specific features and suppress irrelevant features through a semantically guided self-attention mechanism, and injects multimodal cues into each extracted patch token (also called local feature) to enhance class-aware learning at the local level and optimize the final classification effect.

[0023] 4. In the present invention, the model is trained using a global-local self-guided supervision mechanism. On the one hand, global supervision is provided by the Global Average Pooling (GLA) layer. On the other hand, high-quality local self-guided supervision is generated based on semantic information through the same ViT classification model pre-trained on HMP, thereby enhancing the dependency on category-specific labels. That is, by introducing semantically-aware local self-guided supervision on the basis of global supervision to guide patch labeling learning, the generalization performance of the classification model is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0025] Figure 1 This is a framework diagram of a small sample image classification method based on multi-scale cross-modal cue enhancement according to an embodiment of the present invention;

[0026] Figure 2 Schematic diagram of the structure of the feature extractor backbone network used in the embodiment of the present invention;

[0027] Figure 3 This is a visualization of the attention graph obtained by the classification method described in the embodiment of the present invention and the baseline method. DETAILED DESCRIPTION

[0028] It should be noted that the following detailed descriptions are exemplary only and are intended to describe specific embodiments and provide further explanation of the present invention, and are not intended to limit the exemplary embodiments according to the present invention. Unless otherwise indicated, all technical and scientific terms used herein have the same meanings as those commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0029] Explanation of terms

[0030] Small-sample image classification: aims to achieve accurate classification by learning and generalizing to unseen samples using a small number of labeled samples;

[0031] Cross-modal prompts: Combining textual and visual prompts to adjust and enhance the performance of the model in small sample scenarios;

[0032] Vision Transformer / Feature Extractor (ViT): A self-attention-based image recognition model that can capture long-range dependencies and fine-grained features in image classification tasks.

[0033] Example 1

[0034] In order to solve the problems of poor classification performance caused by insufficient cross-modal semantic interaction, global and local supervision limitations, and insufficient spatial robustness in the existing technology, this paper proposes a small-sample image classification method based on multi-scale cross-modal cue enhancement, and innovatively proposes hierarchical multimodal cueing HMP and global-local self-guided supervision GLA mechanism, so as to significantly improve the accuracy and robustness of small-sample image classification.

[0035] This implementation consists of two main components: multi-layered multimodal cues and global-local adaptive supervision. Specifically, textual cues are first encoded using a pre-trained vision-language CLIP model, which extracts rich semantic information from the class names of the supporting image data. Learnable visual cues are then introduced at each scale to adaptively adjust textual cues during training, mitigating cross-modal distribution differences. Furthermore, multimodal cues (MPs) allow the model to align text with visual cues, resulting in unbiased multimodal cues and promoting cross-scale interaction between multimodal cues and visual features. Based on the multi-layered MPs, a hierarchical multimodal cues (HMPs) are constructed to improve spatial robustness. Furthermore, this hierarchical multimodal cues mechanism is introduced into the feature extractor (ViT), enabling the feature extractor (or feature extraction model) to focus on class-specific features and suppress irrelevant ones through a semantically guided self-attention mechanism. Multimodal cues are then injected into each extracted patch label (or local feature) to enhance class-aware learning at the local level. In addition, during the training process, the classification model uses the global average pooling layer (GLA) to provide global supervision, and uses the same ViT model pre-trained by HMP to generate high-quality local self-guided supervision based on semantic information to enhance the dependency on category-specific labels. By further introducing semantic-aware local self-guided supervision on the basis of global supervision to guide patch labeling learning, the generalization performance of the classification model is improved. Using this classification model for small-sample image classification can achieve higher classification accuracy and robust performance.

[0036] This embodiment provides a small sample image classification method based on multi-scale cross-modal cue enhancement, specifically including:

[0037] Obtain a query image and a small sample dataset of support images with labeled categories;

[0038] The query image and the small sample dataset of support images are input into the trained small sample image classification model. The visual features of the query image and the enhanced visual features of each support image are extracted by the feature extractor ViT in the model based on hierarchical multimodal cue enhancement. All the extracted enhanced visual features are globally averaged and pooled to obtain the enhanced visual feature prototype of each category. The category similarity score is calculated based on the visual features of the query image and the enhanced visual feature prototype of each category. The category with the highest score is determined and output as the classification result of the query image.

[0039] The small sample image classification model is trained using a global-local self-guided supervision mechanism.

[0040] The following content introduces the small sample image classification method based on multi-scale cross-modal cue enhancement proposed in this embodiment in more detail.

[0041] First, define the small sample learning problem. Small sample learning aims to learn from the training class C train The knowledge learned in C is generalized to an unseen test class C with limited labeled samples. test ,in This task is usually formalized as an N-way K-shot problem through scenario training, where N represents the number of classes and K represents the number of labeled samples in each class. Each episode (dataset) consists of two parts: a support set S = (x i ,y i ) N×K Contains N×K labeled samples and a query set Q=(x i ,y i ) N×M< Contains N×M test samples for evaluation, where M represents the number of unlabeled samples of each class in the query set. The unlabeled sample is the query image. (x i ,y i ) represents the samples in the support set and query set and their corresponding labels.

[0042] Secondly, if Figure 1 As shown, in this embodiment, a query image to be queried and a small sample dataset of support images with labeled categories are obtained, and these image data are input into a small sample image classification model for classification. The small sample image classification model is equipped with a feature extractor (ViT) based on hierarchical multimodal cue enhancement. The backbone network of the feature extractor ViT is used to extract visual features of the query image, and the feature extractor ViT is also used to extract enhanced visual features of the support images, so as to facilitate the subsequent accurate classification of the small sample images.

[0043] The above-mentioned feature extractor ViT based on hierarchical multimodal cue enhancement includes multiple cascaded feature extraction stages, which are used to extract visual features of different scales of the input image. Each stage is followed by a multimodal cue module, which is used to enhance the visual features extracted in the previous stage using textual cues and learnable visual cues. The multimodal cue modules (MP) of multiple stages constitute a hierarchical multimodal cue (HMP) module.

[0044] (1) Hierarchical Multimodal Prompt (HMP) Module

[0045] The Hierarchical Multimodal Hint Module (HMP) leverages textual and learnable visual cues to enhance supporting features, generating more general class prototypes for classification using a small number of labeled samples. To leverage rich semantic information, textual cues are extracted by inputting the class names of the supporting image annotations into CLIP. The textual cues in these class names serve as prior knowledge for new classes, guiding ViT to extract class-specific supporting features. To mitigate the distributional discrepancy between visual and textual modalities, learnable parameters are introduced as visual cues, which are fused with textual cues during training. Fusion modulates the textual cues to mitigate the inherent distributional discrepancy between the modalities. To enhance robustness to spatial variations, HMP interacts visual features from supporting images at different scales with multimodal cues at different scales, making ViT robust to spatial variations.

[0046] Specifically, such as Figure 1 As shown in , the feature extractor ViT based on layered multimodal cue enhancement is denoted as g, which includes the feature extractor backbone network f (abbreviated as ViT backbone network), as Figure 2 As shown, the ViT backbone network f includes multiple cascaded feature extraction stages and a starting layer set before the multiple cascaded stages; the above-mentioned feature extractor ViT based on hierarchical multimodal prompt enhancement also includes a hierarchical multimodal prompt module set corresponding to each feature extraction stage, that is, a multimodal prompt module is connected after each feature extraction stage. In this embodiment, the ViT backbone network includes three cascaded feature extraction stages and a starting layer set before the three cascaded stages, wherein the starting layer is a convolutional network, the first stage is composed of multiple convolution blocks, the second and third stages are composed of multiple Transformer blocks, and linear projection layers are inserted between each two stages; the Transformer block is composed of a multi-head self-attention layer MSA, two batch normalization layers and a multi-layer perceptron MLP.

[0047] For each labeled category of support image, the support image is input into a feature extractor based on hierarchical multimodal cue enhancement. The three cascade stages of the feature extractor can extract visual features of the support image at different scales. Where i represents the i-th stage, Represents the extracted visual features of the supporting image (can be referred to as supporting features). At the same time, the category annotation label of the supporting image (similar to "A photo of {class}") is input into the text encoder of CLIP to extract the text prompt The text prompt contains rich semantic information. On this basis, multiple groups (three groups in this embodiment) of randomly initialized visual prompt groups of different scales are introduced; each group includes two trainable visual prompts. The scale of each group of visual cues corresponds to the scale of the visual features extracted at each stage. These trainable visual cues can adaptively adjust the text cues to enhance modality compatibility.

[0048] First, the original visual features of the input support image are extracted through the starting layer and the first stage output. The original visual features are combined with the text prompts and the current visual prompts corresponding to the scale of the original visual features, and input into the multimodal prompt module for feature integration to generate the visual features after multimodal prompt integration.

[0049] Specifically, the text prompt is linearly mapped to the input visual features , and at each scale, the mapped text prompts are fused with the visual prompts to obtain the multimodal prompts Z of the corresponding scale mpi , defined as:

[0050]

[0051] Where β is a scaling factor. In the initial stage of model training, the semantic relevance between text and visual cues is limited. In order to enhance the learning of model parameters, β is introduced to scale the visual cues and help parameter learning.

[0052] The two-layer perceptron MLP is used to integrate the input visual features with the multimodal cues of the corresponding scale to obtain the visual features after multimodal cue integration, which is defined as:

[0053]

[0054] Here, MLP stands for Two-Layer Perceptron. By integrating multimodal cues and visual features in the channel dimension, each patch token (or local feature) introduces the global semantic context of multimodal cues, improving the local representation of each patch.

[0055] Secondly, the integrated visual features After being spliced ​​with multimodal cues in the spatial dimension, it is input into the second stage, and the multi-head self-attention mechanism of the Transformer block is used to interact between the input visual features and the multimodal cues to generate cue-enhanced visual features.

[0056] Specific, integrated visual features In the spatial dimension with multimodal cue Z mpi After splicing, the input is sent to the second stage Transformer layer to realize the interaction between patch labels and multimodal cues through the multi-head self-attention mechanism (MSA) to obtain the visual features enhanced by the cues. Defined as:

[0057]

[0058] Among them, || represents the splicing operation; the multi-head attention mechanism MSA refers to mapping each local feature to three vectors Q, K, and V using linear projection, and calculating the attention weight A by the following formula:

[0059]

[0060] The above formula is the self-attention mechanism formula, where d represents the dimension of each attention head. It indicates that the dimension d of each attention head is scaled to prevent the gradient from exploding or disappearing and stabilize the gradient during training.

[0061] By connecting the outputs of all heads and performing linear projection W, the result is obtained and output, which can be expressed as:

[0062]

[0063] Visual features enhanced by the hints obtained in the above way The text prompts and the visual prompts of the corresponding scale are then combined and input into the multimodal prompt module for secondary feature integration using the above feature integration method.

[0064] Finally, the visual features after the second integration are input to the third stage and the multimodal prompt module connected thereto. After the same steps as the second stage and the multimodal prompt module connected thereto, the final prompt-enhanced visual features are extracted and output. Specifically, in this embodiment, the third multimodal prompt mode is actually set between the third and fourth Transformer blocks in the third feature extraction stage. The visual features after the second integration are input to the third stage. After the self-attention mechanism of the third stage and the feature integration of the subsequent multimodal prompt module, the features after the third integration are output. The features are then subjected to the attention mechanism by the fourth Transformer block in the third stage. At this time, the final prompt-enhanced visual features, i.e., enhanced visual features, are output.

[0065] Based on the above approach, by utilizing the same operations at different scales to build a hierarchical structure, ViT can capture different spatial information, thereby enhancing its ability to adapt to spatial changes.

[0066] It should be pointed out here that the support image passes through the above-mentioned feature extractor based on hierarchical multimodal cues to extract the enhanced visual features of the support image; while for the query image, it only passes through the ViT backbone network to extract the visual features of the query image, and does not involve multimodal enhancement processing.

[0067] Considering that the current small sample learning methods are generally divided into two categories: one is to enable the model to quickly adapt to new data through optimization, and the other is to classify by measuring the distance between support samples and query samples, this embodiment adopts the latter classification method, that is, after the above-mentioned feature extractor based on hierarchical multimodal prompts, the visual features of the query image and the enhanced visual features of each support image are extracted, and all the extracted enhanced visual features are input into the global average pooling layer GAP for feature flattening and averaging processing to obtain the enhanced visual feature prototype of each category. At this time, the category similarity score is calculated based on the visual features of the query image and the enhanced visual feature prototype of each category, and the category with the highest score is determined and output as the classification result of the query image.

[0068] Specifically, by averaging the enhanced visual features of all supporting image samples of each class, we can obtain the feature prototypes of N categories, which can be expressed as:

[0069]

[0070] Among them, x j ∈S i , S i represents the i-th support class, |Si| represents the number of support image samples in each support class, g is the feature extractor ViT, c i represents the enhanced visual feature prototype of the i-th support class.

[0071] For the i-th query image q i The visual features f(q i ), by calculating the query image q i Visual feature f(q i ) and N types of enhanced visual feature prototypes c i The cosine distance function between the query image and all support classes determines the similarity score between them. i The probability of belonging to the nth support class is calculated as:

[0072]

[0073] Among them, γ is the temperature parameter, cos() is the cosine distance function, and f represents the ViT backbone network.

[0074] Finally, take the query image q i The classification result is the support class with the highest probability as the final classification result output, so as to achieve accurate classification by extracting generalized class prototypes using only a small number of labeled samples.

[0075] (2) Global-local self-guided supervision mechanism

[0076] In this embodiment, the training of the small sample image classification model is as follows: using the query image of known category and the small sample data set of the support image of the labeled category to train the small sample image classification model, adopting the global-local self-guided supervision mechanism during the training process, and constructing a semantic consistency loss based on visual features. Global supervision loss and local supervision loss The loss function is trained iteratively until the loss function is minimized and the model training is completed. The total loss function can be expressed as:

[0077]

[0078] Among them, λ and α are learning parameters.

[0079] First, for the feature extraction process, a semantic consistency loss based on visual features is constructed.

[0080] Specifically, to alleviate feature bias and model overfitting, consistency loss is performed by calculating the squared deviation between the enhanced visual features and the original visual features. To ensure the overall semantic consistency between the hint-enhanced visual features and the original visual features, the formula is:

[0081]

[0082] Secondly, for global supervision, a prototype-based reasoning method is used, with the cross entropy loss of the output classification result of the query image and the true category label as the global supervision loss.

[0083] Specifically, a cross entropy (CE) loss function is constructed for training, and its formula is:

[0084]

[0085] in, represents the cross entropy (CE) loss, f represents the backbone network of ViT, q i and y i represent the i-th query sample and its true label respectively.

[0086] Finally, to ensure fine-grained feature alignment, in addition to traditional global supervision, patch-level pseudo-labels generated by the same ViT are utilized to guide the patch labeling learning of GLA.

[0087] For local supervision, we use the same structure of the feature extractor ViT based on hierarchical multimodal cue enhancement, combined with the fully connected layer to construct a pseudo classification model, and use the support image small sample dataset to pre-train the pseudo classification model f ls ; Then use the pre-trained pseudo classification model fls The patch-level pseudo-label of the query image is output. The patch-level pseudo-label is the classification confidence score of several local blocks divided in the query image. By generating patch-level pseudo-labels, semantically aware local self-guided supervision can be achieved based on rich semantic information.

[0088] Specifically, through the above method, calculate each local feature z i ∈R L×D The confidence score of is given by:

[0089] f ls (x i )=[z i1 ,z i2 ,…,z iD ];

[0090] Among them, z ij Represents the jth local mark x ij A high confidence score indicates that there is strong semantic consistency between the local feature and the corresponding category, z ij The higher the confidence, the higher the semantic consistency between the local feature and the corresponding class.

[0091] In order to mitigate the influence of irrelevant labels, local blocks with low confidence scores are filtered out and classified as new background classes, and the obtained Specifically, according to the classification confidence score, the top-k sorting method is used to screen out the most relevant k local blocks, and the remaining local blocks are divided into background local blocks; then the dimension of the pseudo label is added. For the background class, the position of the background local block is set to 1, and for the remaining labels (i.e., the most relevant k local blocks), their positions are set to 0. Based on the cross entropy loss of all local blocks in the query image and the background local blocks, the local supervision loss is constructed, and its formula is:

[0092]

[0093] Furthermore, the effectiveness of the method proposed in this embodiment is further evaluated and verified through the following examples on five small sample datasets. Specifically, five small sample datasets are used: the MiniImageNet dataset contains 64 training classes, 16 validation classes, and 20 test classes; the TieredImageNet dataset contains 351 training classes, 97 validation classes, and 160 test classes; the CIFAR-F dataset includes 64 training classes, 16 validation classes, and 20 test classes, using a random partitioning strategy; the CUB-200-2011 (CUB) and Places datasets are used for cross-domain evaluation. In the cross-domain setting, the model is trained on the training set of the MiniImageNet dataset and tested on the test set of the new dataset. Following the common evaluation strategy, the proposed method is evaluated in 5-way 1-shot and 5-way 5-shot settings, with 15 test instances per class. 2,000 episodes are randomly sampled from the test class, and their average accuracy is reported.

[0094] Implementation process: Use Figure 2 The Visformer-T shown is used as the feature extractor backbone network. Compared with ResNet-12, Visformer-T has fewer parameters and is even several times smaller than ViT-S / 16, Swin-T, and WRN-28. The class name template with "A photo of {class}" is encoded by the ViTB / 32CLIP text encoder to produce a 512-dimensional feature vector.

[0095] Training Details: The training process follows the traditional framework, including pre-training and meta-adjustment phases. Input images have a resolution of 224×224 and are processed by Visformer-T, extracting features of sizes 96, 192, and 384. The AdamW optimizer is used with a learning rate of 5e-4 and a cosine learning rate scheduler for regularization. Pre-training is set to 500 epochs with a batch size of 512. During the meta-adjustment phase, the model is trained for 100 epochs using the episodic policy. All experiments were conducted on an NVIDIA GeForce RTX 4090 GPU and implemented in PyTorch.

[0096] To evaluate the effectiveness of our method for common few-shot image classification, experiments were conducted on three benchmark datasets. As shown in Tables 1 and 2 below, our method consistently achieves the best results, outperforming both vision-based and text-based methods in both one-shot and five-shot tasks. Specifically, our method significantly outperforms previous text-based methods (including AM3, SVAE, and SemFew), improving one-shot accuracy by 0.7% and five-shot accuracy by 4.2% on MiniImageNet. Furthermore, compared to vision-based ViT methods (including SUN, FewTURE, and CPEA), our method achieves superior performance with fewer parameters, improving one-shot accuracy by 6.0% and five-shot accuracy by 1.8% on CIFAR-FS. These gains are attributed to the full utilization of semantic information, deep cross-modal interaction through HMP, and the selection of discriminative features through GLA.

[0097] Table 1 Comparison results of this method and other algorithms on MiniImageNet and TieredImageNet (%)

[0098]

[0099] Table 2 Comparison results of this method and other algorithms on CIFAR-FS (%)

[0100]

[0101] To further validate the effectiveness of our method in new tasks for cross-domain few-shot image classification, we tested it on two challenging cross-domain scenarios, including the miniImageNet dataset → CUB and the Places dataset. As shown in Table 3 below, the results demonstrate that our method outperforms its counterparts in all settings. Specifically, our method significantly outperforms previous results, achieving a 3.9% improvement in one-shot accuracy and an 8.9% improvement in five-shot accuracy on CUB. These improvements demonstrate our method's excellent transferability and domain-independence, attributed to its effective capture of discriminative features based on semantic priors, enabling rapid adaptation from training to new datasets. Therefore, even in the presence of domain shift, the proposed classification model generalizes well to new categories with limited labeled samples.

[0102] Table 3 Comparison results of this method and other algorithms on miniImageNet→CUB and Places (%)

[0103]

[0104] like Figure 3The visual attention graph shown in , which intuitively illustrates the effectiveness of the proposed method. The baseline refers to the same ViT model without using the proposed method, as shown in Figure 3 The baseline shown here misclassifies and easily associates other objects with labels in cluttered backgrounds and occlusions. In contrast, guided by semantic information, our method enhances visual features through deep cross-modal interactions and local self-guided supervision, effectively selecting discriminative regions and ignoring irrelevant regions in complex scenes.

[0105] Example 2

[0106] This embodiment provides a small sample image classification system based on multi-scale cross-modal cue enhancement, including:

[0107] A data acquisition module is used to obtain query images and a small sample dataset of support images with annotated categories;

[0108] An image classification module is configured to input a query image and a small sample dataset of support images into a trained small sample image classification model. The model's feature extractor, ViT, based on hierarchical multimodal cue enhancement, extracts the visual features of the query image and the enhanced visual features of each support image. All extracted enhanced visual features are globally averaged and pooled to obtain an enhanced visual feature prototype for each category. A category similarity score is calculated based on the query image visual features and the enhanced visual feature prototypes of each category. The category with the highest score is determined and output as the classification result for the query image.

[0109] The small sample image classification model is trained using a global-local self-guided supervision mechanism.

[0110] Example 3

[0111] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the small sample image classification method based on multi-scale cross-modal cue enhancement are completed as described above.

[0112] Example 4

[0113] This embodiment also provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps in the small sample image classification method based on multi-scale cross-modal cue enhancement are completed as described above.

[0114] The steps involved in the above embodiments 2 to 4 correspond to those in the method embodiment 1. For detailed implementation, please refer to the relevant description of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media that includes one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and cause the processor to perform any method of the present invention.

[0115] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0116] The above description is only a preferred embodiment of the present invention. Although the specific implementation of the present invention is described in conjunction with the accompanying drawings, it does not limit the scope of protection of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A small sample image classification method based on multi-scale cross-modal cue enhancement, characterized by: include: Obtain a query image and a small sample dataset of support images with labeled categories; The query image and the small sample dataset of support images are input into the trained small sample image classification model. The visual features of the query image and the enhanced visual features of each support image are extracted by the feature extractor ViT in the model based on hierarchical multimodal cue enhancement. All the extracted enhanced visual features are globally averaged and pooled to obtain the enhanced visual feature prototype of each category. The category similarity score is calculated based on the visual features of the query image and the enhanced visual feature prototype of each category. The category with the highest score is determined and output as the classification result of the query image. The small sample image classification model is trained using a global-local self-guided supervision mechanism; A small-sample image classification model is trained using a small-sample dataset of query images of known categories and support images of labeled categories. A global-local self-guided supervision mechanism is used during training, and a loss function based on visual feature semantic consistency loss, global supervision loss, and local supervision loss is constructed. The model is trained through continuous iterative training until the loss function is minimized. The visual feature semantic consistency loss is the squared deviation between the original visual features and the enhanced visual features extracted by the feature extractor ViT enhanced based on hierarchical multimodal cues; The global supervision loss is the cross entropy loss based on the output classification result of the query image and the true category label; The construction of the local supervision loss includes: A pseudo-classification model is constructed using a feature extractor ViT enhanced by hierarchical multimodal cues combined with a fully connected layer, and the pseudo-classification model is pre-trained using a small sample dataset of support images. Outputting patch-level pseudo labels of the query image using a pseudo classification model; the patch-level pseudo labels are classification confidence scores of several local blocks divided in the query image; According to the classification confidence score, the top-k sorting method is used to screen out the most relevant k local blocks, and the remaining local blocks are classified as background local blocks; Based on the cross entropy loss of all local patches in the query image and the local patches of background class, a local supervision loss is constructed.

2. The small sample image classification method based on multi-scale cross-modal cue enhancement according to claim 1, characterized in that: The proposed feature extractor ViT based on hierarchical multimodal cue enhancement includes multiple cascaded feature extraction stages, which are used to extract visual features of different scales of the input image. Each stage is followed by a multimodal cue module, which is used to enhance the visual features extracted in the previous stage using textual cues and learnable visual cues.

3. The small sample image classification method based on multi-scale cross-modal cue enhancement according to claim 2, characterized in that: The feature extractor ViT based on layered multimodal cue enhancement includes a ViT backbone network, which includes three cascaded feature extraction stages and a starting layer arranged before the three cascaded stages; wherein the starting layer is a convolutional network, the first stage consists of multiple convolution blocks, the second and third stages each consist of multiple Transformer blocks, and linear projection layers are inserted between each two stages; the Transformer block consists of a multi-head self-attention layer MSA, two batch normalization layers and a multi-layer perceptron MLP; The feature extractor ViT based on hierarchical multimodal prompt enhancement further includes a hierarchical multimodal prompt module respectively provided corresponding to each feature extraction stage, that is, a multimodal prompt module is connected after each feature extraction stage.

4. The small sample image classification method based on multi-scale cross-modal cue enhancement according to claim 3, characterized in that: The extraction of enhanced visual features includes: For each labeled support image, the support image is input into a feature extractor ViT based on hierarchical multimodal cue enhancement. At the same time, the category annotation label of the support image is input into a text encoder to extract text cues. Multiple groups of randomly initialized visual cues of different scales are introduced; each group includes two trainable visual cues, and the scale of each group of visual cues corresponds to the scale of the visual features extracted at each stage. The original visual features of the input support image are extracted through the starting layer and the first stage output. The original visual features are combined with the text prompts and the visual prompts of the current scale and input into the multimodal prompt module for feature integration to generate the visual features after multimodal prompt integration. The integrated visual features are then fed into the second stage, where the multi-head self-attention mechanism of the Transformer block interacts with the input visual features and multimodal cues to generate cue-enhanced visual features. This feature is then combined with the textual cue and the visual cue of the corresponding scale and fed into the multimodal cue module for secondary feature integration. The visual features after secondary integration are then input into the third stage and its connected multimodal prompt module. After the same steps as the second stage and its connected multimodal prompt module, the final prompt-enhanced visual features, i.e., enhanced visual features, are extracted and output.

5. The small sample image classification method based on multi-scale cross-modal cue enhancement according to claim 4, characterized in that: For each scale of visual features, the features are combined with textual and visual cues and fed into the multimodal cue module for feature integration. This generates visual features after multimodal cue integration, including: Linearly map the text prompt to the dimension of the input visual feature, and fuse the mapped text prompt with the visual prompt to obtain a multimodal prompt of the corresponding scale; The two-layer perceptron MLP is used to integrate the input visual features with the multimodal cues of the corresponding scale to obtain the visual features after multimodal cue integration.

6. A small sample image classification system based on multi-scale cross-modal cue enhancement, characterized by: include: A data acquisition module is used to obtain query images and a small sample dataset of support images with annotated categories; An image classification module is configured to input a query image and a small sample dataset of support images into a trained small sample image classification model. The model's feature extractor, ViT, based on hierarchical multimodal cue enhancement, extracts the visual features of the query image and the enhanced visual features of each support image. All extracted enhanced visual features are globally averaged and pooled to obtain an enhanced visual feature prototype for each category. A category similarity score is calculated based on the query image visual features and the enhanced visual feature prototypes of each category. The category with the highest score is determined and output as the classification result for the query image. The small sample image classification model is trained using a global-local self-guided supervision mechanism; A small-sample image classification model is trained using a small-sample dataset of query images of known categories and support images of labeled categories. A global-local self-guided supervision mechanism is used during training, and a loss function based on visual feature semantic consistency loss, global supervision loss, and local supervision loss is constructed. The model is trained through continuous iterative training until the loss function is minimized. The visual feature semantic consistency loss is the squared deviation between the original visual features and the enhanced visual features extracted by the feature extractor ViT enhanced based on hierarchical multimodal cues; The global supervision loss is the cross entropy loss based on the output classification result of the query image and the true category label; The construction of the local supervision loss includes: A pseudo-classification model is constructed using a feature extractor ViT enhanced by hierarchical multimodal cues combined with a fully connected layer, and the pseudo-classification model is pre-trained using a small sample dataset of support images. Outputting patch-level pseudo labels of the query image using a pseudo classification model; the patch-level pseudo labels are classification confidence scores of several local blocks divided in the query image; According to the classification confidence score, the top-k sorting method is used to screen out the most relevant k local blocks, and the remaining local blocks are classified as background local blocks; Based on the cross entropy loss of all local patches in the query image and the local patches of background class, a local supervision loss is constructed.

7. An electronic device, characterized in that: The invention comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of a small sample image classification method based on multi-scale cross-modal cue enhancement as described in any one of claims 1 to 5 are completed.

8. A computer-readable storage medium, characterized in that Used to store computer instructions, which, when executed by a processor, complete the steps of a small sample image classification method based on multi-scale cross-modal cue enhancement as described in any one of claims 1-5.

Images