A small sample image classification method based on multi-granularity semantic prior and semantic guided feature enhancement

Abstract

This invention discloses a few-sample image classification method based on multi-granular semantic priors and semantically guided feature enhancement, belonging to the field of computer image classification. This method constructs a semantic prior for few-sample learning by introducing multi-granular semantic information at both the category and image levels. It then uses category-level semantic similarity to filter and fuse the basic category priors, thereby calibrating the prototype of the new category. Based on this, an image-level semantically guided feature enhancement mechanism is employed to uniformly enhance the visual features of support samples and query samples, improving the consistency and discriminative power of feature representations. This invention can effectively alleviate the prototype estimation bias problem under conditions of extremely small sample sizes, improving the stability and generalization performance of few-sample image classification, and has good application value.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning technology, specifically involving a method for high-precision small-sample image classification by constructing prior knowledge using multi-granular semantic information and guiding visual feature enhancement. Background Technology

[0002] Few-Shot Learning (FSL) aims to enable machines to quickly identify new categories using only a small number of labeled samples, much like humans do. However, existing metric-based or meta-learning methods still face significant challenges in practical applications: First, there is the problem of prototype estimation bias. Because the number of support samples for a new category is extremely small, such as only one or five images, the directly calculated category prototype often has significant bias and fails to accurately reflect the distribution center of the category, leading to a decline in classification performance. Although some research has attempted to utilize data from base classes, effectively filtering and transferring prior knowledge related to the new category remains a challenge. Second, there is insufficient representation of query sample features. Most existing methods focus on how to enhance labeled support samples while neglecting unlabeled query samples. This results in the enhanced support sample features being in a different distribution space than the query sample features, disrupting the consistency of the metric. Furthermore, since the query samples are unknown and unlabeled at the time of testing, directly utilizing their semantic information can easily lead to the risk of label leakage.

[0003] Therefore, there is an urgent need for a method that can effectively utilize prior knowledge of basic categories to calibrate prototypes and synchronously enhance query samples without label leakage. Summary of the Invention

[0004] The purpose of this invention is to provide a few-shot image classification method based on multi-granularity semantic priors and semantically guided feature enhancement. By constructing coarse-grained category-level and fine-grained image-level semantic spaces, it utilizes the maximum a posteriori (MAP) mechanism to calibrate new category prototypes and leverages image description-guided feature enhancement to address the problems of large prototype bias and inconsistent feature distribution under small-shot conditions. In many cases, the method of this invention even surpasses state-of-the-art methods that incorporate unlabeled data, providing new insights for subsequent research and engineering applications of few-shot classification methods based on denoising and feature generation.

[0005] The technical solution adopted in this invention is as follows: A few-sample image classification method based on multi-granularity semantic prior and semantically guided feature enhancement, specifically including the following steps:

[0006] A. Use a pre-trained visual backbone network to extract visual features of support samples, query samples, and basic category samples.

[0007] B. Constructing a multi-granularity semantic feature space: including coarse-grained category-level semantics generated using cue templates, and fine-grained image-level semantics generated using image description generation models;

[0008] C. Construct a basic category prior library: Calculate the visual mean and semantic features of each category in the basic dataset to form a key-value pair library;

[0009] D. MAP Prototype Calibration: During the testing phase, the semantic similarity between the new category and the base category is calculated, and the visual mean of the Top-K similar base categories is retrieved as a prior. The initial prototype of the new category is then calibrated using linear interpolation.

[0010] E. Dual-stream Semantic Guided Feature Enhancement: Design a feature enhancement network that uses image-level semantics as guide (Key / Value) to enhance the attention of visual features (Query). For support samples, aggregated category-level semantics are used to enhance the calibrated prototype; for query samples, their own generated label-independent semantics are used for enhancement.

[0011] F. Calculate the distance between the enhanced prototype and the query features to complete the classification prediction.

[0012] In summary, the present invention has the following beneficial effects:

[0013] First, this method incorporates semantic similarity calculation based on category names, accurately retrieving visual statistical information semantically related to the new category from the base categories. The MAP mechanism effectively corrects initial prototype bias caused by sample scarcity, resulting in more accurate prototype calibration. Second, query samples are augmented with semantic features without leakage. The image description model BLIP is used to directly generate descriptive text from images, independent of real labels. Therefore, it provides rich semantic context for unlabeled query samples without revealing label information, significantly improving the discriminative power of query features. Finally, a symmetric two-stream augmentation architecture is adopted to ensure that the features of both the sample prototype and the query sample are measured within the same augmentation space, improving classification accuracy. Experiments show that this method achieves superior classification accuracy compared to existing techniques on benchmark datasets such as MiniImageNet. This provides new insights for subsequent research and engineering applications of few-shot classification methods based on semantically augmented features. Attached Figure Description

[0014] Figure 1 This is the overall framework diagram of this method;

[0015] Figure 2It is the hyperparameter balance factor in this invention. and neighbor numbers Impact analysis chart on classification accuracy;

[0016] Figure 3 This is a comparison chart showing the effect of the method of the present invention on the feature distribution before and after prototype calibration and feature enhancement, using t-SNE visualization technology.

[0017] Figure 4 This is a comparison chart demonstrating the effectiveness of the method of this invention for image classification using heatmap technology. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0019] We divide the dataset into a base class and a new class, where the sample classes in the new class and the sample classes in the base class are disjoint. When solving the N-way K-shot few-shot classification task, we divide the classes in the new class into a support set and a query set. The support set consists of N randomly sampled classes from the new class, with K samples from each class; that is, the support set has... The query set consists of N samples from these N categories in the new class.

[0020] Reference Figure 1 The method proposed in this invention mainly includes three stages: feature extraction and semantic construction, MAP prototype calibration, and feature enhancement guided by image description text features, including the following steps:

[0021] Step 1: Visual feature extraction using the base dataset Use a pre-trained deep convolutional neural network as a feature extractor For the input image Extract its visual feature vector The formula is as follows:

[0022]

[0023] in, The feature dimension is (e.g., 640 dimensions for ResNet-12 and 768 dimensions for Swin-T).

[0024] Step 2: Constructing multi-granularity semantic features. To utilize semantic information to assist visual classification, this invention constructs semantic features at two granularities, as detailed below:

[0025] Step 201: Coarse-grained category semantics For any category label (Including base category and new category), name its category. Enter the prompt template In China, a pre-trained text encoder using frozen parameters. Feature extraction is performed using the following formula:

[0026] (2)

[0027] This feature is primarily used for subsequent retrieval of the underlying category prior.

[0028] Step 201: Fine-grained image-level semantics For specific image instances (Whether it's a support sample or a query sample), use the pre-trained image description generation model BLIP tool to generate a descriptive text. The formula is as follows:

[0029] (3)

[0030] For example, for a picture of a "sparrow," the generated description might be "a small bird perched on a branch." Then, the same text encoder is used. Feature extraction is performed using the following formula:

[0031] (4)

[0032] This feature is primarily used in the feature enhancement module, and because the generation process does not depend on labels, it is safe for query samples.

[0033] Step 3: Build a basic category prior library. Before meta-testing begins, utilize the basic dataset. Building a prior knowledge base .for Each category Calculate the average visual features of all samples in this category. .knowledge base Store key-value pairs ,in It is a basic category Coarse-grained category semantic features.

[0034] Step 4: MAP Prototype Calibration in the Testing Phase. When facing a new few-shot classification task (N-way K-shot), the first step is to utilize the support set... Calculate the initial prototype To correct for the deviation, perform the following calibration steps:

[0035] Step 401: Similarity Calculation: Calculate the coarse-grained semantics of the current new category. semantics of each basic category in the knowledge base The cosine similarity is calculated using the following formula:

[0036] (5)

[0037] Step 401: Select the one with the highest similarity For each of the 1 base categories, extract their corresponding visual means. The aggregate prior is calculated using the following formula:

[0038] (6)

[0039] Step 402: Prototype calibration using balance factor For example, using 0.33 for 1-shot and 0.7 for 5-shot) for fusion to obtain the calibrated prototype. The calibration formula is as follows:

[0041] (7)

[0042] Step 5: Construct a semantically guided feature enhancement network. This invention designs a general feature enhancement module. This is used to fuse visual and semantic features. The structure is shown in the following steps:

[0043] Step 501: Semantic Projection. Since the visual features (e.g., 640 dimensions) and CLIP text features (512 dimensions) have different dimensions, the semantic features are first projected using a "linear layer + layer normalization + ReLU" module. Mapped to visual space, we get .

[0044] Step 502: Cross-Attention. Based on visual features... The formula for Query is shown below:

[0045] (8)

[0046] semantics after projection For Key ( ) and Value ( The attention level is calculated and weighted, using the following weighting formula:

[0047] (9)

[0048] Step 503: Residual Connection. The attention result is passed through a learnable scaling factor. (Initialized to 0.1) Added back to the original visual features, as shown in the formula below:

[0049] (10)

[0050] Step 6: Dual-stream Enhancement and Classification. Fine-grained image descriptions are used to enhance semantic features for both the support set and query set samples. The specific steps are as follows:

[0051] Step 601: For the new category This enables it to support fine-grained image-level semantics across all samples. Taking the average yields category-level fine-grained semantics. The calibration prototype obtained in step 4 The prototype is obtained by inputting the enhancement network to the prototype enhancement process, as shown in the following formula:

[0052] (11)

[0053] Step 602: Perform semantic augmentation on the query stream samples. For each query image... Directly utilize the image-level semantics it generates and visual features Inputting the data into the augmentation network yields the final query features. The formula is as follows:

[0054] (12)

[0055] Step 603: Classification. Calculation. With all categories of final prototypes The distance between them (such as Euclidean distance or cosine distance) is used to predict the classification result.

[0056] Step 7: Model Training. This invention employs a two-stage training approach. The first stage involves pre-training the visual backbone. The second stage freezes the backbone and trains only the feature enhancement network. .

[0057] The loss function consists of three parts: first, classification loss. The first is the standard cross-entropy loss, which ensures that the enhanced features can be correctly classified. The second is the reconstruction loss. : Force the enhanced features to be as close as possible to the global prototype of the base class To prevent feature drift. Thirdly, supervised contrastive learning loss. By regularizing the image description semantics within the projection space, image description semantics of the same category are encouraged to be closer together, while those of different categories are encouraged to be further apart. The overall loss function formula is shown below:

[0058] (13)

[0059] (14)

[0060] (15)

[0061] (16)

[0062] Experimental validation was performed on four benchmark datasets: MiniImageNet, TieredImageNet, CIFAR-FS, and FC100, using Swin-T and ResNet12 as the backbone for feature extraction. As shown in Table 1, under the 5-way 1-shot setting of MiniImageNet, the proposed method achieved optimal results, significantly outperforming existing metric-based and semantic-based methods. Visualization results ( Figure 3 The results show that after MAP calibration and feature enhancement according to the present invention, the class prototypes are closer to the true center, and the clustering of samples of the same class is more compact. The 5-way 1-shot task and 5-way 5-shot task are constructed on four datasets, and the top-1 average precision and 95% confidence interval of all experiments of more than 600 randomly constructed tasks are given.

[0063] The experimental results on the MiniImageNet and TieredImageNet datasets are compared with state-of-the-art methods. The comparison results are shown in Table 2 below:

[0064] Table 1

[0065]

[0066] The experimental results on the CIFAR-FS and FS100 datasets are compared with state-of-the-art methods. The comparison results are shown in Table 2 below:

[0067] Table 2

[0068]

[0069] This invention provides a comprehensive comparison with state-of-the-art methods, listing the backbone network and semantic information of each method for clear comparison. Results for these methods are cited from their original works or relevant references. As shown in Tables 1 and 2, the method of this invention achieves state-of-the-art performance on MiniImageNet, TieredImageNet, and FC100 datasets. Due to the small image size and low resolution of CIFAR-FS images, the semantic quality of image descriptions generated using the BLIP model is not high, resulting in a suboptimal performance on this dataset.

[0070] Furthermore, to verify the effectiveness of this invention, the t-SNE method was used to reduce the dimensionality of the feature vectors of samples in a certain task query set in a classification task and project them onto a two-dimensional plane, such as... Figure 3 As shown, after our MAP-calibrated prototype and semantic feature enhancement, the overall clustering is more compact and the predicted prototype is closer to the true center of the query set.

Claims

1. A few-sample image classification method based on multi-granularity semantic prior and semantically guided feature enhancement, characterized in that, The specific steps are as follows: A. Obtain supporting samples and query samples from the task to be classified, and extract the corresponding visual features through a pre-trained visual feature extraction network. B. Construct multi-granularity semantic features based on the support samples and query samples, including category-level semantic features and image-level semantic features; C. Based on the visual features of the basic category samples, calculate the statistical prior information of each basic category and construct the basic category prior library; D. During the testing phase, the semantic similarity between the new category and the base category is calculated using the category-level semantic features. Prior information related to the semantics of the new category is selected from the prior library of the base category, and the initial prototype of the new category is calibrated. E. Employ a feature enhancement network based on image-level semantic information to perform unified semantic-guided feature enhancement on the visual features of the calibrated support sample prototype and the query sample. F. Calculate the similarity between the enhanced support sample prototype and the query sample features, and output the classification result of the query sample.

2. The few-sample image classification method based on multi-granularity semantic prior and semantically guided feature enhancement according to claim 1, characterized in that: The specific steps for extracting the corresponding visual features using a pre-trained visual feature extraction network in step A above are as follows: A1. Utilizing a pre-trained visual backbone network Extract input image Original visual features The formula is as follows:

3. The few-sample image classification method based on multi-granularity semantic prior and semantically guided feature enhancement according to claim 1, characterized in that: The specific steps for constructing multi-granularity semantic features based on support samples and query samples in step B above are as follows: B1. Construct category-level coarse-grained semantic features: for any category label Its natural language name Embedded into the prompt template "a photo of ", and through the frozen text encoder Obtaining category semantic features The formula is as follows: B2. Constructing image-level fine-grained semantic features: Utilizing image description generation model tools Directly from the input image Generate descriptive sentences and through the text encoder Encoding as instance-level semantic features The formula is as follows: The instance-level semantic features are generated without the need for real labels.

4. The few-sample image classification method based on multi-granularity semantic prior and semantically guided feature enhancement according to claim 1, characterized in that: The steps described in step C above, which involve calculating the statistical prior information of each basic category based on the visual features of the basic category samples and constructing a basic category prior library, are as follows: C1. Utilizing visual feature extraction networks Extracting the base dataset Each basic category The visual features of all samples are analyzed, and their visual mean is calculated. ; C2. Combining semantic features corresponding to the basic categories Construct a basic category prior library containing visual statistical information and semantic information. The formula is as follows:

5. The few-sample image classification method based on multi-granularity semantic prior and semantically guided feature enhancement according to claim 1, characterized in that: The specific steps for calculating semantic similarity using category-level semantic features and calibrating the initial prototype of the new category in step D above are as follows: D1. For a new category in the current task, calculate its coarse-grained category semantics. With the basic category prior library semantics of each basic category Cosine similarity between The formula is as follows: D2. Based on similarity Sort the basic categories and select the top-K basic categories with the highest similarity to form a neighbor set. And retrieve the corresponding visual mean. To compute aggregate priors The formula is as follows: D3. Based on the maximum a posteriori probability principle, introduce a balanced hyperparameter. Utilizing aggregated priors For the support set The initial prototype obtained by calculation Perform linear interpolation calibration to obtain the calibrated prototype. The formula is as follows: in, Used to adjust the weights of observational data and prior knowledge.

6. The few-sample image classification method based on multi-granularity semantic prior and semantically guided feature enhancement according to claim 1, characterized in that: In step E, the feature enhancement network used is based on a multi-head cross-attention mechanism, wherein the number of attention heads is... Set to 4; by linearly projecting the feature vectors into multiple subspaces and computing attention scores in parallel, and then concatenating and linearly transforming the outputs of each head, the association information between image-level semantics and visual features in different dimensions is captured. The specific construction steps are as follows: E1, Feature Alignment: The input auxiliary semantic context is aligned using a non-linear projection module. Mapping to the visual feature space yields the projected semantic features. The formula is as follows: in, and For the weights and biases of the linear layer, Representation layer normalization; E2. Perform multi-head cross-attention enhancement. The calibration prototype obtained in step 4... (or query features) Using the query vector as the reference, its corresponding image-level semantic features are... After linear projection, they become the key and value vectors. Specifically, to enhance the diversity of feature extraction, this invention employs a multi-head mechanism. The calculation process is as follows: the Query, Key, and Value are respectively processed through... The learnable weight matrix is ​​projected onto different subspaces (in this embodiment) In each subspace, the dot product similarity between the Query and the Key is calculated, and after Softmax normalization, the weight coefficients are obtained and applied to the Value vector; The outputs of each head are concatenated; after passing through a linear projection layer and residual connections, the final enhanced feature vector is obtained. Compared to single-head attention, the multi-head cross-attention mechanism allows the model to focus on information from different parts of the image description semantics in different representation subspaces, thereby generating more discriminative feature representations. Cross-attention computation will visual input As a query, the projected semantic features As keys and values, an attention mechanism is constructed, and its formula is as follows: The semantic awareness attention map is calculated and visual features are recalibrated using the following formula: E3, Residual Fusion: Introducing a Learnable Scaling Factor The attention output is added to the original visual features to obtain the enhanced features. The formula is as follows: E4. Support for sample prototype enhancement: For each new category First, aggregate the support set for this category. The instance-level semantic features of all samples form a category-level semantic context. The formula is as follows: The calibration prototype obtained in step D As a visual anchor point, combined with the category-level semantic context Through feature enhancement networks Generate the final prototype The formula is as follows: E5. Query Sample Feature Enhancement: For each query image Directly utilize the instance-level semantics it generates Its visual features Enhancement is performed to obtain the final query features. The formula is as follows:

7. The few-sample image classification method based on multi-granularity semantic prior and semantically guided feature enhancement according to claim 1, characterized in that: During the model training phase, a total loss function, which includes classification loss and reconstruction loss, is constructed and optimized, as follows: F1, Classification Loss The cross-entropy loss is calculated based on the enhanced features to ensure the discriminative power of the features, and is expressed by the following formula: F2, Reconstruction Loss : The global prototype of the constrained features and the base categories Alignment, to prevent feature drift, is expressed by the following formula: F3. Considering that the text generated by the image description model may contain noise, this invention introduces supervised contrastive learning loss. By regularizing the image description semantics within the projection space, we encourage image description semantics of the same category to be close together and those of different categories to be far apart, thereby improving the system's robustness to fluctuations in the quality of the descriptive text. The calculation steps and parameter definitions are as follows:

1. Similarity measurement: Calculate the image description feature vector of anchor samples within a batch using cosine similarity. Compared with the feature vectors of the remaining samples 1) Semantic alignment distance between them; 2) Probability distribution mapping: Introducing a temperature adjustment factor The semantic alignment distance is scaled and then calculated through exponentialization and normalization to obtain the predicted probability of the anchor sample matching each sample in the batch; 3) Supervised consistency constraint: Construct a set of positive samples based on the category label information of the samples. 4) The loss function is defined as follows: Calculate the negative log average of the probability of matching the anchor sample with all positive samples of the same class; in, For training batch size, For the current anchor point sample index, For positive sample index, This represents the cosine similarity operation between vectors. F4, Total Loss It can be expressed by the following formula:

Images