Small sample image classification method and system based on multi-scale cross-modal prompt enhancement

Abstract

The invention discloses a small sample image classification method and system based on multi-scale cross-modal prompt enhancement, and relates to the technical field of small sample image classification, and the method comprises the steps: obtaining a query image and a support image small sample data set with a labeled category; inputting the acquired data into a trained small sample image classification model, extracting visual features of a query image and enhanced visual features of each support image through a feature extractor based on hierarchical multi-modal prompt enhancement in the model, and performing global average pooling on all the extracted enhanced visual features to obtain an enhanced visual feature of each support image; obtaining an enhanced visual feature prototype of each category, calculating similarity scores of the categories according to the visual features of the query image and the enhanced visual feature prototype of each category, determining the category with the highest score as a classification result of the query image, and outputting the classification result; wherein the small sample image classification model is obtained by training a global-local self-guiding supervision mechanism. According to the invention, the precision and robustness of small sample image classification can be significantly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of small sample image classification, and in particular to a small sample image classification method and system based on multi-scale cross-modal cue enhancement. Background Art

[0002] At present, deep learning has achieved great success in computer vision fields such as image classification. However, traditional deep learning-based image classification methods usually rely heavily on large-scale labeled datasets to achieve high classification accuracy, and these datasets and their annotation costs are high and sometimes even impractical. For example, in rare disease diagnosis and industrial anomaly detection, the amount of image data is extremely large, and it is difficult to label each image data separately. On the contrary, humans can recognize new objects from only a few pictures. In order to imitate human learning ability and reduce dependence on annotated data, existing studies have proposed a few-shot learning method (Few-Shot Learning, FSL) that learns from a limited amount of data. This method is used to achieve few-shot image classification, that is, to learn from a small number of labeled samples and generalize to unseen samples, so as to achieve accurate classification of unseen sample images. Although the above methods have made significant progress, it is still challenging to achieve human-level performance, especially in the 1-shot scenario (i.e., single-shot learning scenario) with only one labeled sample. Therefore, for few-shot image classification, it is necessary to further optimize the few-shot learning method to improve the overall classification performance.

[0003] To improve classification performance, existing small sample learning methods have proposed using Transformer layers as classifiers. However, this classification model often has difficulty generating robust and general visual representations with limited support data. To this end, it has also been proposed to enhance visual features by using text modality as auxiliary information through adaptive mechanisms, such as fusing text and visual prototypes, or learning classifiers using graph convolutional networks. However, due to the inherent distribution differences between modalities and the lack of deep cross-modal interactions in the underlying features, the proposed method has difficulty capturing complex cross-modal semantic relationships.

[0004] At the same time, this text-based method often only focuses on global supervision, ignoring local semantic modeling and fine-grained discriminative information. Semantic information modeling in local features helps avoid learning bias and non-generalizable features, thereby significantly improving the generalization ability of the model, which is crucial for small sample learning. Although existing vision-based methods also try to guide local feature learning through local supervision, they often face semantic confusion due to the lack of class-specific semantic information, making it difficult to improve supervision at specific locations.

[0005] In addition, existing small-sample learning methods lack the ability to process visual features at multiple scales, resulting in poor performance when dealing with different spatial changes. This lack of a hierarchical structure makes it difficult for the classification model to remain robust in the face of scale and position changes, which will also lead to a reduction in overall classification performance. Summary of the invention

[0006] To address the deficiencies of the above-mentioned prior art, the present invention provides a small sample image classification method and system based on multi-scale cross-modal cue enhancement, proposes a hierarchical multimodal cue (HMP) and a global-local self-guided supervision (GLA) mechanism, makes full use of cross-modal information, and conducts deep cross-modal interaction through the hierarchical multimodal cue mechanism to capture complex cross-modal semantic relationships, which are used to enhance visual features, and further improves the robustness to spatial changes through a hierarchical structure. At the same time, through the global-local self-guided supervision mechanism, global and local discriminative features are selected to generate a universal and generalized visual representation. The above method significantly improves the accuracy and robustness of small sample image classification.

[0007] In a first aspect, the present invention provides a small sample image classification method based on multi-scale cross-modal cue enhancement.

[0008] A small sample image classification method based on multi-scale cross-modal cue enhancement, comprising:

[0009] Get a query image and a small sample dataset of supporting images with labeled categories;

[0010] The query image and the small sample data set of the support images are input into the trained small sample image classification model. The visual features of the query image and the enhanced visual features of each support image are extracted by the feature extractor ViT based on hierarchical multimodal cue enhancement in the model. After all the extracted enhanced visual features are globally averaged and pooled, the enhanced visual feature prototype of each category is obtained. The similarity score of the category is calculated based on the visual features of the query image and the enhanced visual feature prototype of each category, and the category with the highest score is determined and output as the classification result of the query image.

[0011] The small sample image classification model is trained using a global-local self-guided supervision mechanism.

[0012] In a second aspect, the present invention provides a small sample image classification system based on multi-scale cross-modal cue enhancement.

[0013] A small sample image classification system based on multi-scale cross-modal cue enhancement, comprising:

[0014] A data acquisition module, used to obtain query images and a small sample dataset of supporting images with annotated categories;

[0015] An image classification module is used to input the query image and the small sample data set of the support images into the trained small sample image classification model, extract the visual features of the query image and the enhanced visual features of each support image through the feature extractor ViT based on hierarchical multimodal cue enhancement in the model, obtain the enhanced visual feature prototype of each category after all the extracted enhanced visual features are globally averaged pooled, calculate the category similarity score based on the query image visual features and the enhanced visual feature prototype of each category, determine the category with the highest score, and output it as the classification result of the query image;

[0016] The small sample image classification model is trained using a global-local self-guided supervision mechanism.

[0017] In a third aspect, the present invention further provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps of the method described in the first aspect are completed.

[0018] In a fourth aspect, the present invention further provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the method described in the first aspect.

[0019] One or more of the above technical solutions have the following beneficial effects:

[0020] 1. The present invention provides a small sample image classification method and system based on multi-scale cross-modal cue enhancement, proposes a hierarchical multimodal cue HMP and a global-local self-guided supervision GLA mechanism, makes full use of cross-modal information, and performs deep cross-modal interaction through a hierarchical multimodal cue mechanism to capture complex cross-modal semantic relationships, which are used to enhance visual features, and further improve the robustness to spatial changes through a hierarchical structure. At the same time, through a global-local self-guided supervision mechanism, global and local discriminative features are selected, and thereby label dependency learning guided by semantic information is accelerated to generate general and generalized visual representations. The above method significantly improves the accuracy and robustness of small sample image classification, and solves the problems of poor classification performance caused by insufficient cross-modal semantic interaction, global and local supervision limitations, and insufficient spatial robustness in the prior art.

[0021] 2. The present invention adopts a hierarchical multimodal prompt HMP mechanism. In each layer of the multimodal prompt MP, a pre-trained visual-language CLIP model is used to encode the category text prompt to extract rich semantic information from the category name. Then, at each scale of the extracted multi-scale visual features, a learnable visual prompt is introduced to adaptively adjust the text prompt during the training process to alleviate the distribution difference of cross-modal prompts. At the same time, the multimodal prompt MP allows the model to align the text prompt with the visual prompt when fusing the text prompt and the visual prompt, thereby generating unbiased multimodal prompts and promoting the interaction between cross-scale multimodal prompts and visual features. Through a multi-layer multimodal prompt MP structure, a hierarchical multimodal prompt HMP module that can improve spatial robustness is constructed.

[0022] 3. The present invention introduces a hierarchical multimodal cueing mechanism in the feature extractor / model (Vision Transformer, ViT), so that the feature extractor can focus on class-specific features and suppress irrelevant features through a self-attention mechanism guided by semantic information, and inject multimodal cues into each extracted patch token (also called local feature) to enhance class-aware learning at the local level and optimize the final classification effect.

[0023] 4. In the present invention, the model is trained using a global-local self-guided supervision mechanism. On the one hand, global supervision is provided by the global average pooling layer (GLA). On the other hand, high-quality local self-guided supervision is generated based on semantic information through the same ViT classification model pre-trained based on HMP, thereby enhancing the dependency specific to category labels. That is, semantically-aware local self-guided supervision is introduced on the basis of global supervision to guide patch labeling learning, thereby effectively improving the generalization performance of the classification model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0025] Figure 1 This is a framework diagram of a small sample image classification method based on multi-scale cross-modal cue enhancement according to an embodiment of the present invention;

[0026] Figure 2 Schematic diagram of the structure of the feature extractor backbone network used in the embodiment of the present invention;

[0027] Figure 3 The visual attention map obtained by the classification method described in the embodiment of the present invention and the baseline method. DETAILED DESCRIPTION

[0028] It should be noted that the following detailed descriptions are exemplary only, are intended to describe specific embodiments, are intended to provide further explanation of the present invention, and are not intended to limit exemplary embodiments according to the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art to which the present invention belongs. In addition, it should also be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0029] Explanation of terms

[0030] Small sample image classification: aims to achieve accurate classification by learning and generalizing to unseen samples with a small number of labeled samples;

[0031] Cross-modal prompts: Combine textual prompts and visual prompts to adjust and enhance the performance of the model in small sample scenarios;

[0032] Vision Transformer / Feature Extractor (ViT): A self-attention based image recognition model that can capture long-range dependencies and fine-grained features in image classification tasks.

[0033] Embodiment 1

[0034] In order to solve the problems of poor classification performance caused by insufficient cross-modal semantic interaction, global and local supervision limitations, and insufficient spatial robustness in the prior art, this paper proposes a small sample image classification method based on multi-scale cross-modal cue enhancement, and innovatively proposes a hierarchical multimodal cue HMP and a global-local self-guided supervision GLA mechanism, so as to significantly improve the accuracy and robustness of small sample image classification.

[0035] This embodiment mainly consists of two parts: multi-layer multimodal prompts and global-local adaptive supervision. Specifically, the text prompts are first encoded by the pre-trained visual-language CLIP model to extract rich semantic information from the class names of the supporting image data; then, learnable visual prompts are introduced at each scale to adaptively adjust the text prompts during the training process to alleviate the cross-modal distribution differences; moreover, the multimodal prompts MP allow the model to align the text with the visual prompts, thereby generating unbiased multimodal prompts and promoting the interaction between multimodal prompts and visual features across scales; based on the multi-layer multimodal prompts MP, a hierarchical multimodal prompt (HMP) is constructed to improve spatial robustness. On this basis, the above-mentioned hierarchical multimodal prompt mechanism is introduced in the feature extractor (ViT), so that the feature extractor (or feature extraction model) can focus on class-specific features and suppress irrelevant features through a self-attention mechanism guided by semantic information, and multimodal prompts are injected into each extracted patch mark (or local feature) to enhance class-aware learning at the local level. In addition, during the training process of the classification model, the global average pooling layer (GLA) is used to provide global supervision, and the same ViT model pre-trained by HMP is used to generate high-quality local self-guided supervision based on semantic information to enhance the dependency on category-specific tags. By further introducing semantically-aware local self-guided supervision on the basis of global supervision to guide patch labeling learning, the generalization performance of the classification model is improved. Using this classification model for small-sample image classification can achieve higher classification accuracy and robust performance.

[0036] The present embodiment provides a small sample image classification method based on multi-scale cross-modal cue enhancement, specifically including:

[0037] Get a query image and a small sample dataset of supporting images with labeled categories;

[0038] The query image and the small sample data set of the support images are input into the trained small sample image classification model. The visual features of the query image and the enhanced visual features of each support image are extracted by the feature extractor ViT based on hierarchical multimodal cue enhancement in the model. After all the extracted enhanced visual features are globally averaged and pooled, the enhanced visual feature prototype of each category is obtained. The similarity score of the category is calculated based on the visual features of the query image and the enhanced visual feature prototype of each category, and the category with the highest score is determined and output as the classification result of the query image.

[0039] The small sample image classification model is trained using a global-local self-guided supervision mechanism.

[0040] The following content introduces the small sample image classification method based on multi-scale cross-modal cue enhancement proposed in this embodiment in more detail.

[0041] First, define the small sample learning problem. Small sample learning aims to learn from the training class C train The knowledge learned in C is generalized to unseen test classes C with limited labeled samples. test ,in This task is usually formalized as an N-way K-shot problem through scenario training, where N represents the number of classes and K represents the number of labeled samples in each class. Each episode (dataset) consists of two parts: a support set S = (x i ,y i ) N×K Contains N×K labeled samples and a query set Q=(x i ,y i ) N×M< Contains N×M test samples for evaluation, where M represents the number of unlabeled samples of each category in the query set. The unlabeled samples are the query images. i ,y i ) represents the samples in the support set and query set and their corresponding labels.

[0042] Secondly, if Figure 1 As shown, in this embodiment, a query image to be queried and a small sample data set of support images with annotated categories are obtained, and these image data are input into the small sample image classification model for classification. Among them, a feature extractor (ViT) based on hierarchical multimodal prompt enhancement is provided in the small sample image classification model, and the backbone network of the feature extractor ViT is used to extract the visual features of the query image, and the feature extractor ViT is used to extract the enhanced visual features of the support image, so as to facilitate the subsequent accurate classification of the small sample image.

[0043] The above-mentioned feature extractor ViT based on hierarchical multimodal cue enhancement includes multiple cascaded feature extraction stages, which are used to extract visual features of different scales of the input image, and a multimodal cue module is provided after each stage to enhance the visual features extracted in the previous stage by using text cues and learnable visual cues, wherein the multimodal cue modules (MP) of multiple stages constitute a hierarchical multimodal cue (HMP) module.

[0044] (1) Hierarchical Multimodal Prompt (HMP) Module

[0045] The Hierarchical Multimodal Hint Module uses textual cues and learnable visual cues to enhance support features and generate more general class prototypes for classification with a small number of labeled samples. In order to utilize rich semantic information, textual cues are extracted by inputting the class name of the support image annotation in CLIP, and the textual cues in the class name of the support image annotation are used as the prior knowledge of the new class to guide ViT to extract class-specific support features; in order to alleviate the distribution difference between visual and textual modalities, learnable parameters are introduced as visual cues, which are used to fuse with textual cues during training, and the textual cues are adjusted through fusion to alleviate the inherent distribution difference between modalities; in order to enhance the robustness to spatial changes, HMP interacts the visual features of the support images of different scales with the multimodal cues of different scales, so that ViT is robust to spatial changes.

[0046] Specifically, Figure 1 As shown in FIG, the feature extractor ViT based on layered multimodal cue enhancement is denoted as g, which includes a feature extractor backbone network f (referred to as ViT backbone network for short), as Figure 2 As shown, the ViT backbone network f includes multiple cascaded feature extraction stages and a starting layer set before the multiple cascaded stages; the above-mentioned feature extractor ViT based on hierarchical multimodal prompt enhancement also includes a hierarchical multimodal prompt module set corresponding to each feature extraction stage, that is, a multimodal prompt module is connected after each feature extraction stage. In this embodiment, the ViT backbone network includes three cascaded feature extraction stages and a starting layer set before the three cascaded stages, wherein the starting layer is a convolutional network, the first stage is composed of multiple convolution blocks, the second and third stages are composed of multiple Transformer blocks, and a linear projection layer is inserted between the two stages; the Transformer block is composed of a multi-head self-attention layer MSA, two batch normalization layers and a multi-layer perceptron MLP.

[0047] For each labeled class of support images, the support images are fed into a feature extractor enhanced with hierarchical multimodal cues. The three cascaded stages of the feature extractor can extract visual features of the support images at different scales. Where i represents the i-th stage, Represents the extracted visual features of the supporting image (can be referred to as supporting features for short). At the same time, the category annotation label of the supporting image (similar to "A photo of {class}") is input into the text encoder of CLIP to extract the text prompt The text prompt contains rich semantic information. On this basis, multiple groups (three groups in this embodiment) of randomly initialized visual prompt groups of different scales are introduced; each group includes two trainable visual prompts The scale of each group of visual cues corresponds to the scale of the visual features extracted in each stage. These trainable visual cues can adaptively adjust the text cues to enhance the compatibility of the modalities.

[0048] First, the input support image is extracted through the original visual features of the starting layer and the first stage output. The original visual features are combined with the text prompts and the current visual prompts corresponding to the scale of the original visual features, and input into the multimodal prompt module for feature integration to generate the visual features after multimodal prompt integration.

[0049] Specifically, the text prompt is linearly mapped to the input visual features , and at each scale, the mapped text prompts are fused with the visual prompts to obtain the multimodal prompts Z of the corresponding scale mpi , defined as:

[0050]

[0051] Among them, β is a scaling factor. In the initial stage of model training, the semantic correlation between text and visual cues is limited. In order to enhance the learning of model parameters, β is introduced to scale the visual cues and help parameter learning.

[0052] The two-layer perceptron MLP is used to integrate the input visual features with the multimodal cues of the corresponding scale to obtain the visual features after the multimodal cues are integrated, which is defined as:

[0053]

[0054] Here, MLP stands for two-layer perceptron. By integrating multimodal cues and visual features in the channel dimension, each patch token (or local feature) introduces the global semantic context information of multimodal cues and improves the local representation of each patch.

[0055] Secondly, the integrated visual features After being concatenated with multimodal cues in the spatial dimension, it is input into the second stage, where the multi-head self-attention mechanism of the Transformer block interacts with the input visual features and multimodal cues to generate cue-enhanced visual features.

[0056] Specific, integrated visual features In the spatial dimension, Z mpi After concatenation, the images are fed into the second-stage Transformer layer to implement the interaction between patch labels and multimodal cues through the multi-head self-attention mechanism (MSA) to obtain visual features enhanced by cues. Defined as:

[0057]

[0058] Among them, || represents the splicing operation; the multi-head attention mechanism MSA refers to mapping each local feature to three vectors Q, K, and V using linear projection, and calculating the attention weight A through the following formula:

[0059]

[0060] The above formula is the self-attention mechanism formula, where d represents the dimension of each attention head. It indicates that the dimension d of each attention head is scaled to prevent the gradient from exploding or disappearing and stabilize the gradient during training.

[0061] By connecting the outputs of all heads and performing linear projection W, the result is obtained and output, which can be expressed as:

[0062]

[0063] Visual features enhanced by hints obtained in the above manner The text prompts and the visual prompts of the corresponding scale are then combined and input into the multimodal prompt module for secondary feature integration using the above feature integration method.

[0064] Finally, the visual features after the second integration are input to the third stage and the multimodal prompt module connected thereto, and after the same steps as the second stage and the multimodal prompt module connected thereto, the final prompt-enhanced visual features are extracted and output. Specifically, in this embodiment, the third multimodal prompt mode is actually set between the third and fourth Transformer blocks in the third feature extraction stage, and the visual features after the second integration are input to the third stage, and after the feature integration of the self-attention mechanism of the third stage and the subsequent multimodal prompt module, the features after the third integration are output, and the features are then passed through the fourth Transformer block in the third stage to perform the attention mechanism, and the final prompt-enhanced visual features, i.e., enhanced visual features, are output.

[0065] Based on the above approach, by utilizing the same operations at different scales and building a hierarchical structure, ViT can capture different spatial information, thereby enhancing its ability to adapt to spatial changes.

[0066] It should be pointed out here that the support image is subjected to the above-mentioned feature extractor based on hierarchical multimodal cues to extract the enhanced visual features of the support image; while for the query image, it only passes through the ViT backbone network to extract the visual features of the query image, and it does not involve multimodal enhancement processing.

[0067] Considering that the current small sample learning methods are generally divided into two categories: one is to make the model quickly adapt to new data through optimization, and the other is to classify by measuring the distance between the support sample and the query sample. The latter classification method is adopted in this embodiment, that is, after the feature extractor based on hierarchical multimodal prompts is used, the visual features of the query image and the enhanced visual features of each support image are extracted, and all the extracted enhanced visual features are input into the global average pooling layer GAP for feature flattening and averaging to obtain the enhanced visual feature prototype of each category. At this time, according to the visual features of the query image and the enhanced visual feature prototype of each category, the similarity score of the category is calculated, and the category with the highest score is determined and output as the classification result of the query image.

[0068] Specifically, the feature prototypes of N categories are obtained by averaging the enhanced visual features of all supporting image samples of each class, which can be expressed as:

[0069]

[0070] Among them, x j ∈S i , S i represents the i-th support class, |Si| represents the number of support image samples in each support class, g is the feature extractor ViT, c i represents the enhanced visual feature prototype of the i-th support class.

[0071] For the i-th query image q i The visual features f(q i ), by calculating the query image q i Visual feature f(q i ) and N types of enhanced visual feature prototypes c i The cosine distance function between the query image and all supporting classes determines the similarity score between them. i The probability of belonging to the nth support class is calculated as:

[0072]

[0073] Among them, γ is the temperature parameter, cos() is the cosine distance function, and f represents the ViT backbone network.

[0074] Finally, take the query image q i The classification result is the support class with the highest probability as the final classification result output, so as to achieve accurate classification by extracting generalized class prototypes using only a small number of labeled samples.

[0075] (2) Global-local self-guided supervision mechanism

[0076] In this embodiment, the training of the small sample image classification model is as follows: using the query image of known category and the small sample data set of the support image of labeled category to train the small sample image classification model, adopting the global-local self-guided supervision mechanism during the training process, and constructing a semantic consistency loss based on visual features. Global supervision loss and local supervision loss The loss function is trained continuously iteratively until the loss function is minimized to complete the model training. The total loss function can be expressed as:

[0077]

[0078] Among them, λ and α are learning parameters.

[0079] First, for the feature extraction process, a semantic consistency loss based on visual features is constructed.

[0080] Specifically, to alleviate feature bias and model overfitting, a consistency loss is performed by calculating the squared deviation between the enhanced visual features and the original visual features. To ensure the overall semantic consistency between the visual features enhanced by the hint and the original visual features, the formula is:

[0081]

[0082] Secondly, for global supervision, a prototype-based reasoning method is used, with the cross entropy loss of the output classification result of the query image and the true category label as the global supervision loss.

[0083] Specifically, a minimization cross entropy (CE) loss function is constructed for training, and its formula is:

[0084]

[0085] in, represents the cross entropy (CE) loss, f represents the backbone network of ViT, and q i and i They represent the i-th query sample and its true label respectively.

[0086] Finally, to ensure fine-grained feature alignment, in addition to traditional global supervision, patch-level pseudo-labels generated by the same ViT are utilized to guide the patch labeling learning of GLA.

[0087] For local supervision, we use the same structure of feature extractor ViT based on hierarchical multimodal cue enhancement, combined with a fully connected layer to construct a pseudo classification model, and use the small sample dataset of support images to pre-train the pseudo classification model f ls ; Then use the pre-trained pseudo classification model fls The patch-level pseudo-label of the query image is output. The patch-level pseudo-label is the classification confidence score of several local blocks divided in the query image. By generating patch-level pseudo-labels, semantically-aware local self-guided supervision can be achieved based on rich semantic information.

[0088] Specifically, through the above method, each local feature z is calculated i ∈R L×D The confidence score of is:

[0089] f ls (x i )=[z i1 ,z i2 ,…,z iD ];

[0090] Among them, z ij Represents the jth local label x ij A high confidence score indicates that there is a strong semantic consistency between the local feature and the corresponding category. ij The higher the confidence, the higher the semantic consistency between the local feature and the corresponding class.

[0091] In order to mitigate the impact of irrelevant labels, local blocks with low confidence scores are filtered out and classified as new background classes. Specifically, according to the classification confidence score, the top-k sorting method is used to screen out the most relevant k local blocks, and the remaining local blocks are divided into background local blocks; then the dimension of the pseudo-label is added, for the background class, the position of the background local block is set to 1, and for the remaining labels (i.e., the most relevant k local blocks), their positions are set to 0, and then the local supervision loss is constructed based on the cross entropy loss of all local blocks in the query image and the background local blocks. The formula is:

[0092]

[0093] Furthermore, the effectiveness of the method proposed in this embodiment is further evaluated and verified by the following examples on five small sample data sets. Specifically, five small sample data sets are used: the MiniImageNet data set contains 64 training classes, 16 validation classes, and 20 test classes; the TieredImageNet data set contains 351 training classes, 97 validation classes, and 160 test classes; the CIFAR-F data set includes 64 training classes, 16 validation classes, and 20 test classes, using a random partitioning strategy; the CUB-200-2011 (CUB) and Places data sets are used for cross-domain evaluation, where in the cross-domain setting, the model is trained on the training set of the MiniImageNet data set and tested on the test set of the new data set. According to the common evaluation strategy, the proposed method is evaluated in the 5-way 1-shot and 5-way5-shot settings, with 15 test instances in each category. 2,000 episodes are randomly selected from the test class, and their average accuracy is reported.

[0094] Implementation process: Use Figure 2 The Visformer-T shown is used as the feature extractor backbone network. Compared with ResNet-12, Visformer-T has fewer parameters and is even several times smaller than ViT-S / 16, Swin-T, and WRN-28; the class name template with "A photoof{class}" is encoded by the ViTB / 32CLIP text encoder to produce a 512-dimensional feature vector.

[0095] Training details: The training process follows the traditional framework, including pre-training and meta-adjustment stages. The input image has a resolution of 224×224 and is processed by Visformer-T to extract features of size 96, 192, and 384. The AdamW optimizer is used with a learning rate of 5e-4 and is adjusted using a cosine learning rate scheduler. The pre-training epoch is set to 500 epochs and the batch size is 512. In the meta-adjustment stage, the model is trained for 100 epochs using the scenario policy. All experiments are conducted on an NVIDIA GeForce RTX 4090 GPU and implemented in PyTorch.

[0096] To evaluate the effectiveness of our method for common small sample image classification, experiments were conducted on three benchmark datasets. As shown in Tables 1 and 2 below, our method consistently achieves the best results, outperforming both vision-based and text-based methods in both 1-shot and 5-shot tasks. Specifically, our method significantly outperforms previous text-based methods (including AM3, SVAE, and SemFew methods), improving the 1-shot accuracy by 0.7% and the 5-shot accuracy by 4.2% on MiniImageNet. In addition, compared with vision-based ViT methods (including SUN, FewTURE, and CPEA), our method achieves superior performance with fewer parameters, improving the 1-shot accuracy by 6.0% and the 5-shot accuracy by 1.8% on CIFAR-FS. These gains are attributed to making full use of semantic information, achieving deep cross-modal interaction through HMP, and selecting discriminative features through GLA.

[0097] Table 1 Comparison results of this method and other algorithms on MiniImageNet and TieredImageNet (%)

[0098]

[0099] Table 2 Comparison results of this method and other algorithms on CIFAR-FS (%)

[0100]

[0101] For cross-domain small sample image classification, to further verify the effectiveness of this method in new tasks, we tested it in two challenging cross-domain scenarios, including the miniImageNet dataset → CUB and the Places dataset. As shown in Table 3 below, the results show that this method outperforms its counterparts in all settings. Specifically, the performance of this method is significantly better than the previous results, with a 3.9% improvement in 1-shot accuracy and an 8.9% improvement in 5-shot accuracy on CUB. These improvements demonstrate the excellent transferability and domain-independence of this method, which is attributed to the effective capture of discriminative features based on semantic priors, enabling rapid adaptation from training to new datasets. Therefore, even in the presence of domain shift, the classification model proposed by this method can be well generalized to new categories with limited labeled samples.

[0102] Table 3 Comparison results of this method with other algorithms in miniImageNet→CUB and Places (%)

[0103]

[0104] like Figure 3The visual attention graph shown in Figure 1 intuitively illustrates the effectiveness of the proposed method. The baseline refers to the same ViT model without the proposed method, as shown in Figure 1. Figure 3 The shown baselines misclassify and easily associate other objects with labels in cluttered backgrounds and occlusions. In contrast, guided by semantic information, our method enhances visual features through deep cross-modal interactions and local self-guided supervision, thus effectively selecting discriminative regions and ignoring irrelevant regions in complex scenes.

[0105] Embodiment 2

[0106] This embodiment provides a small sample image classification system based on multi-scale cross-modal cue enhancement, including:

[0107] A data acquisition module, used to obtain query images and a small sample dataset of supporting images with annotated categories;

[0108] An image classification module is used to input the query image and the small sample data set of the support images into the trained small sample image classification model, extract the visual features of the query image and the enhanced visual features of each support image through the feature extractor ViT based on hierarchical multimodal cue enhancement in the model, obtain the enhanced visual feature prototype of each category after all the extracted enhanced visual features are globally averaged pooled, calculate the category similarity score based on the query image visual features and the enhanced visual feature prototype of each category, determine the category with the highest score, and output it as the classification result of the query image;

[0109] The small sample image classification model is trained using a global-local self-guided supervision mechanism.

[0110] Embodiment 3

[0111] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the small sample image classification method based on multi-scale cross-modal cue enhancement as described above are completed.

[0112] Embodiment 4

[0113] This embodiment also provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps in the small sample image classification method based on multi-scale cross-modal cue enhancement as described above are completed.

[0114] The steps involved in the above embodiments 2 to 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0115] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0116] The above description is only a preferred embodiment of the present invention. Although the specific implementation mode of the present invention is described in conjunction with the accompanying drawings, it is not a limitation of the protection scope of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A small sample image classification method based on multi-scale cross-modal cue enhancement, characterized in that: include: Get a query image and a small sample dataset of supporting images with labeled categories; The query image and the small sample data set of the support images are input into the trained small sample image classification model. The visual features of the query image and the enhanced visual features of each support image are extracted by the feature extractor ViT based on hierarchical multimodal cue enhancement in the model. After all the extracted enhanced visual features are globally averaged and pooled, the enhanced visual feature prototype of each category is obtained. The similarity score of the category is calculated based on the visual features of the query image and the enhanced visual feature prototype of each category, and the category with the highest score is determined and output as the classification result of the query image. The small sample image classification model is trained using a global-local self-guided supervision mechanism.

2. The small sample image classification method based on multi-scale cross-modal cue enhancement as claimed in claim 1, characterized in that: The hierarchical multimodal cue-enhanced feature extractor ViT comprises a plurality of cascaded feature extraction stages for extracting visual features of different scales of an input image, and a multimodal cue module is provided after each stage for enhancing the visual features extracted in the previous stage with textual cues and learnable visual cues.

3. The small sample image classification method based on multi-scale cross-modal cue enhancement as claimed in claim 2, characterized in that: The feature extractor ViT based on layered multimodal cue enhancement includes a ViT backbone network, which includes three cascaded feature extraction stages and a starting layer arranged before the three cascaded stages; wherein the starting layer is a convolutional network, the first stage is composed of multiple convolution blocks, the second and third stages are both composed of multiple Transformer blocks, and linear projection layers are inserted between the two stages; the Transformer block is composed of a multi-head self-attention layer MSA, two batch normalization layers and a multi-layer perceptron MLP; The feature extractor ViT based on hierarchical multimodal prompt enhancement also includes a hierarchical multimodal prompt module respectively arranged corresponding to each feature extraction stage, that is, a multimodal prompt module is connected after each feature extraction stage.

4. The small sample image classification method based on multi-scale cross-modal cue enhancement as claimed in claim 3, characterized in that: The extraction of enhanced visual features comprises: For each labeled category of support images, the support images are input into the feature extractor ViT based on hierarchical multimodal cue enhancement. At the same time, the category annotation labels of the support images are input into the text encoder to extract text cues, and multiple groups of randomly initialized visual cues of different scales are introduced; each group includes two trainable visual cues, and the scale of each group of visual cues corresponds to the scale of the visual features extracted at each stage; The original visual features extracted from the input support image through the starting layer and the first stage output are combined with the text prompt and the visual prompt of the current scale, and input into the multimodal prompt module for feature integration to generate the visual features after multimodal prompt integration; The integrated visual features are then input into the second stage, where the multi-head self-attention mechanism of the Transformer block interacts with the input visual features and the multimodal prompts to generate prompt-enhanced visual features. The features are then combined with text prompts and visual prompts of the corresponding scale and input into the multimodal prompt module for secondary feature integration. The visual features after the secondary integration are then input into the third stage and its connected multimodal prompt module, and after the same steps as the second stage and its connected multimodal prompt module, the final prompt-enhanced visual features, i.e., enhanced visual features, are extracted and output.

5. The small sample image classification method based on multi-scale cross-modal cue enhancement as claimed in claim 4, characterized in that: For each scale of visual features, the features are combined with textual prompts and visual prompts and input into the multimodal prompt module for feature integration to generate visual features after multimodal prompt integration, including: Linearly map the text prompt to the dimension of the input visual feature, and fuse the mapped text prompt with the visual prompt to obtain a multimodal prompt of the corresponding scale; The two-layer perceptron MLP is used to integrate the input visual features with the multimodal cues of the corresponding scale to obtain the visual features after the multimodal cues are integrated.

6. The small sample image classification method based on multi-scale cross-modal cue enhancement as claimed in claim 1, characterized in that: A small sample image classification model is trained using a small sample dataset of query images of known categories and support images of labeled categories. A global-local self-guided supervision mechanism is adopted during the training process, and a loss function based on visual feature semantic consistency loss, global supervision loss and local supervision loss is constructed. The model training is completed through continuous iterative training until the loss function is minimized.

7. The small sample image classification method based on multi-scale cross-modal cue enhancement as claimed in claim 6, characterized in that: The visual feature semantic consistency loss is the squared deviation between the original visual features and the enhanced visual features extracted by the feature extractor ViT enhanced based on hierarchical multimodal cues; The global supervision loss is a cross entropy loss based on the output classification result of the query image and the true category label; The construction of the local supervision loss includes: A pseudo classification model is constructed using a feature extractor ViT based on hierarchical multimodal cue enhancement combined with a fully connected layer, and the pseudo classification model is pre-trained using a small sample dataset of support images. Outputting a patch-level pseudo label of the query image using a pseudo classification model; the patch-level pseudo label is a classification confidence score of a plurality of local blocks divided in the query image; According to the classification confidence score, the top-k sorting method is used to screen out the most relevant k local blocks, and the remaining local blocks are classified as background local blocks; Based on the cross entropy loss of all local patches in the query image and the local patches of background classes, a local supervision loss is constructed.

8. A small sample image classification system based on multi-scale cross-modal cue enhancement, characterized in that: include: A data acquisition module, used to obtain query images and a small sample dataset of supporting images with annotated categories; An image classification module is used to input the query image and the small sample data set of the support images into the trained small sample image classification model, extract the visual features of the query image and the enhanced visual features of each support image through the feature extractor ViT based on hierarchical multimodal cue enhancement in the model, obtain the enhanced visual feature prototype of each category after all the extracted enhanced visual features are globally averaged pooled, calculate the category similarity score based on the query image visual features and the enhanced visual feature prototype of each category, determine the category with the highest score, and output it as the classification result of the query image; The small sample image classification model is trained using a global-local self-guided supervision mechanism.

9. An electronic device, characterized in that: The invention comprises a memory and a processor and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of a small sample image classification method based on multi-scale cross-modal cue enhancement as described in any one of claims 1 to 7 are completed.

10. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of a small sample image classification method based on multi-scale cross-modal cue enhancement as described in any one of claims 1-7.

Images