A small sample image recognition method based on class perception gate and multi-branch fusion

Abstract

This invention relates to the field of image processing, specifically a few-shot image recognition method based on class-aware gating and multi-branch fusion. First, a few-shot image recognition task is performed by sampling from the training set (N-way K-shot). Second, image features are extracted and category features are generated. Third, based on the category features, local features are dynamically weighted and modulated using a class-aware gating mechanism to enhance category-related information. Then, a first and second metric process are performed in parallel to calculate the similarity between the support sample and the query sample, namely a local similarity metric calculated using a dense similarity convolution branch and a global similarity metric calculated using a class-aware Mahalanobis distance branch, respectively. Finally, the two similarity metrics are adaptively weighted and fused using learnable fusion weights with Sigmoid constraints to determine the query sample category. This invention effectively improves classification accuracy and generalization ability under few-shot conditions and is suitable for few-shot learning tasks in image recognition and object detection.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a few-sample image recognition method based on class-aware gating and multi-branch fusion. Background Technology

[0002] Deep learning technology has made significant progress in image recognition fields (such as facial recognition, autonomous driving, and robot vision), primarily due to the support of large-scale datasets and powerful computing resources. However, this technology heavily relies on a large amount of high-quality labeled data, and in practical applications, data acquisition and labeling face multiple constraints. For example, biometric data involving personal privacy (such as facial data) is strictly restricted by laws and regulations, while public concerns about data security are increasing. On the other hand, the scarcity of samples in certain special scenarios (such as rare protected animals, special industrial defects, or rare medical images) also makes it difficult to adequately train models. Furthermore, data labeling typically requires specialized knowledge and a large amount of manpower, resulting in high costs and long cycles, severely restricting the widespread application of deep learning in image recognition tasks.

[0003] In contrast, the human visual system exhibits powerful few-shot learning capabilities, enabling it to quickly identify new objects using only a very small number of samples. Inspired by this, few-shot learning has emerged, aiming to allow models to rapidly generalize to new categories using only a small number of labeled samples, building upon a foundation of already learned known categories. Related methods include meta-learning, metric learning, data augmentation, and transfer learning, and have become important research directions in computer vision and machine learning, showing broad application prospects in sample-scarce scenarios such as medical image analysis, autonomous driving, and special object detection.

[0004] In methods for solving the problem of few-sample image recognition, metric learning-based strategies are widely used due to their effectiveness. The core idea is to learn a highly discriminative feature embedding space, such that samples of the same class are close together while samples of different classes are far apart, thereby achieving rapid recognition and generalization of new categories.

[0005] Traditional methods typically rely on image-level global semantic features for representation. However, in few-shot learning, training data is extremely limited, and global features often fail to capture fine-grained visual information, leading to the loss of discriminative local features and thus affecting the accuracy of similarity measurement.

[0006] To alleviate this problem, researchers in recent years have tended to use local feature combinations to represent images and designed various alignment mechanisms to match local regions between query set and support set images to achieve more refined similarity comparisons. Although these methods, which focus on local feature alignment, have shown superior performance on multiple benchmark datasets, they still have significant drawbacks: on the one hand, existing methods mostly focus on improving the feature alignment quality between the query and support sets, while ignoring the inter-class discriminative information contained in the samples within the support set; on the other hand, current alignment mechanisms often over-match local regions that are irrelevant to class judgment, introducing noise and interfering with the accuracy of similarity measurements. Summary of the Invention

[0007] The purpose of this invention is to provide a small sample image recognition method based on class-aware gating and multi-branch fusion, which mainly solves the problems existing in the prior art. It can significantly improve classification accuracy and generalization ability under the condition of only a small number of labeled samples.

[0008] To achieve the above objectives, the technical solution adopted by this invention is to provide a few-sample image recognition method based on class-aware gating and multi-branch fusion, characterized by comprising the following steps:

[0009] Step S100: Sample a small sample image recognition task from the training set, including a support set and a query set; the support set contains classified support samples, while the query set contains query samples to be classified.

[0010] Step S200: Extract image features from the support set and the query set, including global features and local features; simultaneously, generate category features for each category based on the global features of similar samples in the support set;

[0011] Step S300: Based on the category features, an adaptive modulation weight is generated through a class-aware gating mechanism to dynamically weight and modulate the local features of the support set and the query set to enhance the information related to the category.

[0012] Step S400: The first and second measurement processes are performed in parallel to calculate the similarity between the support samples and the query samples. The first measurement process performs a local feature measurement process, which calculates the dense local matching similarity between the query samples and the support samples based on the modulated local features, as the first similarity measure. The second measurement process performs a global feature measurement process, which constructs statistical distribution models for each category based on the modulated features of the same type of support samples, and calculates the matching degree between the query sample features and the statistical distribution models of each category, as the second similarity measure.

[0013] Step S500: Based on the fusion result of the first similarity measure and the second similarity measure, determine the category of the query sample.

[0014] Further, in step S300, the local features are first adapted, and then the category features are fused with the local features through a class-aware gating mechanism. Modulation weights are generated through a neural network to weight the local features.

[0015] Furthermore, the feature adaptation comprises two steps: first, the local features from the support set and the query set are adapted by a multilayer perceptron to obtain adapted local features, thereby enhancing nonlinear expressive power; then, the adapted local features are subjected to layer normalization to maintain numerical stability.

[0016] Furthermore, the feature is that the class-aware gating mechanism first concatenates the class feature with a local feature to form a concatenated feature; then, the concatenated feature is input into a gating generation network, which outputs a gating value in the interval [0, λmax]; finally, the gating value is used to weight the class feature, and the weighted class feature is added element-wise to the local feature to obtain the adapted local feature.

[0017] Further, in step S400, the first measurement process is a dense similarity convolution branch. First, the similarity between each modulated local feature block of the query sample and each modulated local feature block of the support sample is calculated to form a dense similarity matrix. Then, the similarity matrix is ​​processed by a convolutional neural network to extract spatial matching patterns, and the first similarity metric is obtained by global average pooling.

[0018] Furthermore, the feature is that the pairwise similarity is cosine similarity; and the convolutional neural network is a lightweight network.

[0019] Further, in step S400, the second metric process is a class-aware Mahalanobis distance branch. First, based on the modulated features of the support samples of the same class, the mean vector and covariance matrix of the class are calculated to construct the statistical distribution model; then, the Mahalanobis distance from the features of the query sample to the statistical distribution model of each class is calculated and converted into the second similarity metric.

[0020] Furthermore, the covariance matrix is ​​a contracted diagonal covariance matrix.

[0021] Further, in step S500, the fusion result is obtained by calculating a learnable weighted parameter, that is, setting a learnable parameter β, constraining its value range by the Sigmoid function, and using the formula:

[0022] Final classification score = β × first similarity measure + (1-β) × second similarity measure.

[0023] Furthermore, the local features are extracted through a backbone network, and the local features are image patch token feature sequences; the category features are the mean of the class token features of each category of supporting samples; the backbone network is a visual Transformer, or a combination of a visual Transformer and a convolutional neural network.

[0024] In view of the above technical features, the few-sample image recognition method based on class-aware gating and multi-branch fusion of the present invention has the following significant advantages compared with the prior art:

[0025] 1. This invention is a few-sample image recognition method based on class-aware gating and multi-branch fusion. By fusing dense similarity convolution and class-aware Mahalanobis distance into a dual-branch structure, it effectively combines local detail matching and global distribution modeling, significantly improving the accuracy and robustness of few-sample classification.

[0026] 2. The present invention is a few-sample image recognition method based on class-aware gating and multi-branch fusion. By introducing a class-aware gating mechanism, key features are dynamically enhanced and redundant information is suppressed, thereby strengthening the model's ability to extract discriminative features and its generalization performance. Attached Figure Description

[0027] Figure 1 This is a system block diagram of a preferred embodiment of the few-sample image recognition method based on class-aware gating and multi-branch fusion of the present invention;

[0028] Figure 2 This is a flowchart of a preferred embodiment of the small sample image recognition method based on class-aware gating and multi-branch fusion of the present invention. Detailed Implementation

[0029] The present invention will be further described below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0030] Please see Figure 1 and Figure 2This invention discloses a few-sample image recognition method based on class-aware gating and multi-branch fusion. As shown in the figure, a preferred embodiment includes the following steps:

[0031] Step S1, Task Construction.

[0032] A specific few-shot learning task is constructed from a large-scale basic training set using the N-way K-shot paradigm. Specifically, N classes are randomly selected from the basic training set, and K images are randomly drawn from each class. These K images have class labels (i.e., they explicitly indicate which class they belong to), and they constitute the support set. Then, from the remaining images of the same N classes, Q images are drawn from each class. These Q images do not have valid class labels, i.e., they do not indicate which class they belong to, and they constitute the query set. The goal of this few-shot image recognition method is to accurately predict the labels of all samples (i.e., N × Q images) in the query set using the information provided by the K images in the support set.

[0033] Step S2: Feature extraction and category feature generation.

[0034] Using a backbone network, features are extracted from all images (N × (K + Q) images) in the support and query sets, yielding two key outputs: global features and local features. The backbone network employs a self-supervised pre-trained Visual Transformer (ViT-S / 16) as the feature extractor. Input images are segmented into fixed-size blocks and embedded as sequences. Global features are represented by the [CLS] token in the ViT output sequence, which aggregates the global semantic information of the image. Local features are represented by all image block tokens in the ViT output sequence, excluding [CLS], which encode local detail information of the image. For images in the support set, they are classified according to their category. The average of the global features of the K images in each category is used as the category feature to label that category, serving as the semantic guidance signal for subsequent class-aware modulation.

[0035] In scenarios with few samples, single-label annotation can lead to "supervision collapse" because it guides the model to focus on features that are only useful for distinguishing training classes, rather than general features with good transferability. To address this issue, a self-supervised approach is used for model pre-training to improve the model's generalization ability.

[0036] In other embodiments, the backbone network may also be a combination of a visual Transformer and a convolutional neural network.

[0037] Step S3, Feature Adaptation and Class-Aware Modulation.

[0038] This step utilizes class features from the support set to adaptively enhance the class-related portions of the local features in both the support and query sets. Specifically, feature adaptation is first performed on all samples in both sets, by inputting the original local features of all samples from both sets into a multilayer perceptron (MLP). This MLP typically consists of two fully connected layers with a ReLU activation function in between, used to perform a non-linear transformation on the features to enhance their expressive power. The features output by the MLP are then layer-normalized to stabilize training and prevent numerical fluctuations. The adapted and normalized local features are then used as the input for the next step of class-aware gating modulation.

[0039] In class-aware gating modulation, for each sample to be modulated (whether in the support set or the query set), the class features are concatenated to its adapted local features to form a joint feature vector. Samples in the support set have prior known classification information, so they are concatenated with known class features. Samples in the query set do not have prior classification information, so only their own global features are concatenated.

[0040] The concatenated feature vector is input into a gated generative network, specifically a two-layer multilayer perceptron with GELU activation. Its output is a scalar gate value. This scalar gate value is mapped to the (0,1) interval using a sigmoid function, yielding the basic gate weights. To allow for stronger information injection, the basic gate weights are multiplied by an adjustable maximum threshold value λ_max, resulting in the final gate coefficients. These gate coefficients are related to the content of the image patch. For example, background regions typically receive smaller gate coefficients, while foreground object regions receive larger ones. Subsequently, the class features are weighted using the gate coefficients, i.e., the element-wise product of the gate coefficients and class features is calculated to obtain weighted class information. The weighted class information is then added element-wise to the corresponding local features to generate adapted local features. This mechanism enables adaptive enhancement of local features by class features, making support samples and query samples more discriminative at the class level.

[0041] Step S4, dual-branch similarity measurement.

[0042] After obtaining the adaptively enhanced support set features and query set features, two complementary similarity calculations are performed in parallel in this step: one for matching from a local detail perspective and the other for matching from a global perspective.

[0043] The first measurement process employs a dense similarity convolution branch to obtain a first similarity score based on local detail matching. Specifically, a dense similarity matrix is ​​first calculated. For each query sample, the cosine similarity between its adapted image patch features and the features of each supporting class prototype image patch is calculated, forming a dense similarity matrix. Next, this dense similarity matrix is ​​input into a lightweight convolutional neural network. This convolutional neural network typically consists of three convolutional layers using small-sized kernels (e.g., 3×3) to obtain a single-channel two-dimensional response map. Finally, global average pooling is performed on the feature map output by the convolutional neural network, compressing it into a scalar value. This scalar is the first similarity score for the query sample based on local detail matching, i.e., the first similarity measure.

[0044] The second metric employs a class-aware Mahalanobis distance branch to obtain a second similarity score based on global distribution matching. Specifically, a statistical model is first constructed using the modulated local features of its K support samples, and the class mean vector and diagonal covariance matrix are calculated. Then, based on the modulated features of the query sample (using the same class as the support samples to guide modulation), attention is requested using the class mean. A weighted average aggregation is used to obtain the global representation of the query sample in that class, and the Mahalanobis distance between the global representation and the class mean is calculated. Finally, the Mahalanobis distance is converted into a similarity score: the negative squared distance is taken as the second similarity score (i.e., the second similarity metric) for that class, with a smaller distance resulting in a higher score.

[0045] Step S5, adaptive fusion classification.

[0046] For each query sample relative to a specific category, the first and second similarity metrics are fused using a learnable fusion weight to obtain the final classification score. The fusion weight is obtained as follows: a learnable scalar parameter β is defined in the model, and during actual fusion, it is constrained to the (0,1) interval using a sigmoid function. The model automatically learns the optimal value of the scalar parameter β during training. Based on the scalar parameter β, the final classification score is obtained by the following formula:

[0047] Final classification score = β × first similarity measure + (1-β) × second similarity measure.

[0048] For each query sample, calculate its final classification score relative to all N categories. The category corresponding to the maximum value of the fused final classification score is taken as the predicted classification result for that query sample.

[0049] The training of this method employs a meta-learning paradigm, which is completely consistent with steps S1 to S5 during inference. First, a large number of N-way K-shot tasks (episodes) are repeatedly and randomly sampled from a large base dataset (such as miniImageNet). Then, for each task, the complete process of S1 to S5 is executed. In each training iteration, the model needs to learn to recognize these N categories using the limited labeled examples provided by the support set, and ultimately complete the classification prediction of the query set samples. During training, the prediction loss of the model for all samples in the query set is first calculated (e.g., using the cross-entropy loss function). Then, the gradient of the loss with respect to all learnable parameters in the model (including some parameters of the ViT backbone, MLP weights, gated network weights, convolutional kernel weights, shrinkage coefficients, fusion parameters, modulation intensity, etc.) is calculated using the backpropagation algorithm. These parameters are updated using an optimizer (such as Adam).

[0050] Through training on massive tasks, the model learns not features specific to a particular category, but rather how to use "class-aware gating" to purify features and how to balance the two metrics of "local matching" and "global distribution" to achieve a general ability for fast classification, thereby enabling it to generalize to entirely new categories.

[0051] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A few-sample image recognition method based on class-aware gating and multi-branch fusion, characterized in that, Includes the following steps: Step S100: Sample a small sample image recognition task from the training set, including a support set and a query set; the support set contains classified support samples, while the query set contains query samples to be classified. Step S200: Extract image features from the support set and the query set, including global features and local features; simultaneously, generate category features for each category based on the global features of similar samples in the support set; Step S300: Based on the category features, an adaptive modulation weight is generated through a class-aware gating mechanism to dynamically weight and modulate the local features of the support set and the query set to enhance the information related to the category. Step S400: The first and second measurement processes are performed in parallel to calculate the similarity between the support samples and the query samples. The first measurement process performs a local feature measurement process, which calculates the dense local matching similarity between the query samples and the support samples based on the modulated local features, as the first similarity measure. The second measurement process performs a global feature measurement process, which constructs statistical distribution models for each category based on the modulated features of the same type of support samples, and calculates the matching degree between the query sample features and the statistical distribution models of each category, as the second similarity measure. Step S500: Based on the fusion result of the first similarity measure and the second similarity measure, determine the category of the query sample.

2. The few-sample image recognition method based on class-aware gating and multi-branch fusion according to claim 1, characterized in that, In step S300, the local features are first adapted, and then the category features are fused with the local features through a class-aware gating mechanism. Modulation weights are generated through a neural network to weight the local features.

3. The few-sample image recognition method based on class-aware gating and multi-branch fusion according to claim 2, characterized in that, The feature adaptation includes two steps: first, the local features from the support set and the query set are adapted by a multilayer perceptron to obtain adapted local features, so as to enhance the nonlinear expressive power; then, the adapted local features are normalized by layers to maintain numerical stability.

4. The few-sample image recognition method based on class-aware gating and multi-branch fusion according to claim 2, characterized in that, The class-aware gating mechanism first concatenates the class feature with a local feature to form a concatenated feature; then, the concatenated feature is input into a gating generation network, which outputs a gating value in the interval [0, λmax]; finally, the gating value is used to weight the class feature, and the weighted class feature is added element-wise to the local feature to obtain the adapted local feature.

5. The few-sample image recognition method based on class-aware gating and multi-branch fusion according to claim 1, in step S400, the first measurement process is a dense similarity convolution branch, firstly calculating the similarity between each modulated local feature block of the query sample and each modulated local feature block of the support sample to form a dense similarity matrix; then performing convolutional neural network processing on the similarity matrix to extract spatial matching patterns, and obtaining the first similarity metric through global average pooling.

6. The few-sample image recognition method based on class-aware gating and multi-branch fusion according to claim 5, characterized in that, The pairwise similarity is cosine similarity; the convolutional neural network is a lightweight network.

7. The small sample image recognition method based on class-aware gating and multi-branch fusion according to claim 1, in step S400, the second metric process is a class-aware Mahalanobis distance branch. First, based on the features of the support samples of the same class after modulation, the mean vector and covariance matrix of the class are calculated to construct the statistical distribution model; then, the Mahalanobis distance from the features of the query sample to the statistical distribution model of each class is calculated and converted into the second similarity metric.

8. The few-sample image recognition method based on class-aware gating and multi-branch fusion according to claim 7, characterized in that, The covariance matrix is ​​a contracted diagonal covariance matrix.

9. The few-sample image recognition method based on class-aware gating and multi-branch fusion according to claim 1, characterized in that, In step S500, the fusion result is obtained by calculating a learnable weighted parameter, that is, setting a learnable parameter β, constraining its value range by the Sigmoid function, and using the formula: Final classification score = β × first similarity measure + (1-β) × second similarity measure.

10. The few-sample image recognition method based on class-aware gating and multi-branch fusion according to claim 1, characterized in that, The local features are extracted through a backbone network, and the local features are image block token feature sequences; the category features are the mean of the class token features of each category of supporting samples; the backbone network is a visual Transformer, or a combination of a visual Transformer and a convolutional neural network.

Images