A small sample image classification method fusing relationship selection and multi-scale distribution

Abstract

The application discloses a small sample image classification method fusing relationship selection and multi-scale distribution. 1, data set acquisition and establishment, support set and query set in the training stage, and auxiliary set in the test stage; 2, adopting Conv-64F as a feature extractor, initial feature pairs are obtained; 3, the initial feature pairs are respectively input into a local relationship selection measurement branch and a global feature distribution measurement branch; preliminary prediction scores I and II of the query image are obtained; 4, the obtained preliminary prediction scores I and II of the query image are combined and then input into an adaptive fusion layer, and final scores of the query image belonging to various categories are obtained; 5, the category corresponding to the maximum score is the predicted category, and a cross-entropy loss function is calculated with the real sample image category corresponding to the query image, and model parameters are updated through gradient back propagation. The model disclosed by the application effectively improves the small sample learning accuracy.

Description

Technical Field

[0001] This invention relates to the field of few-shot learning, and in particular to a few-shot image classification method that integrates relation selection and multi-scale distribution. Background Technology

[0002] With the advent of the big data era, deep learning models have achieved advanced results in tasks such as image classification and text classification. However, the success of deep learning models largely depends on a large amount of training data. In real-world scenarios, some categories have only a small amount of data or a small amount of labeled data, and labeling unlabeled data would consume a lot of time and manpower. In contrast, humans are very good at recognizing a new object with very few samples. For example, a child only needs a few pictures in a book to recognize what a "tiger" and an "elephant" are. Inspired by human rapid learning ability, researchers hope that machine learning models, after learning a large amount of data for a certain category, can quickly learn for new categories with only a small number of samples. This is the problem that Few-Shot Learning aims to solve.

[0003] Currently, few-shot learning is mainly divided into three categories: transfer learning, metric learning, and meta-learning. Transfer learning has been widely applied in many fields; it involves fine-tuning a pre-trained network using a certain amount of labeled data. Metric learning methods primarily model the distance between samples, ensuring that similar samples are compact and dissimilar samples are far apart in the metric space. Meta-learning methods aim to create a model that can adapt to multi-task learning scenarios. For few-shot learning tasks, since the input categories are different each time, each training task can be treated as a different learning task, thus allowing meta-learning methods to be used for model training.

[0004] The core idea of ​​metric-based few-sample image classification is to directly measure the relationship between the query image and the support images, thereby learning transferable feature embeddings. Recent works have shown that metric learning based on local features provides richer representations than image-level metrics. For example, Li et al. proposed a metric learning approach that combines local feature metrics and global feature distribution metrics with query and support features, and compared to instance-level metrics, it can more richly represent the similarity information between the query image and support samples. However, this local feature-based metric learning suffers from the problem of consistent relational importance, failing to highlight the similarity of important regions among numerous local features and suppressing the similarity of unimportant regions. Furthermore, the global distribution metric branch only obtains feature distribution information from the original features; this single-scale feature distribution information cannot effectively represent the statistical information of class distribution.

[0005] This invention proposes a new model structure that enables the selection of local feature relationships and a more representative global feature distribution. Summary of the Invention

[0006] This invention provides a few-sample image classification method that integrates relation selection and multi-scale distribution. The method mainly consists of two parts. The first part designs a novel module for relation selection based on local features, adaptively weighting important relations and suppressing trivial relations. The second part generates a multi-scale feature distribution based on the original features to measure the feature distribution of the classes, fusing the distribution information of features at different scales to alleviate the problem of unrepresentativeness of a single-scale feature distribution. Experimental results show that this method has higher accuracy than the benchmark.

[0007] A few-sample image classification method that integrates relation selection and multi-scale distribution, comprising the following steps:

[0008] Step (1): Dataset acquisition and creation, using the public dataset MiniImageNet. In the few-shot learning scenario, the dataset is divided into three sets: the support set and query set for the training phase, and the auxiliary set for the testing phase.

[0009] Step (2): Using Conv-64F as a feature extractor, input the RGB image into the feature extractor to obtain the initial feature pairs F. q ,F s .

[0010] Step (3): Pair the initial features with F q ,F s The inputs are respectively fed into the local relation selection metric branch and the global feature distribution metric branch. The local relation selection metric branch uses the relation selection module to autonomously weight and sum the local relations between the query image and the supporting classes to obtain the preliminary prediction score I of the query image; the global feature distribution metric branch measures the difference in distribution between the query image and the supporting classes based on the statistical information of the local features to obtain the preliminary prediction score II of the query image.

[0011] Step (4): The preliminary prediction scores I and II of the query image obtained under the two strategies in step (3) are merged and then input into the adaptive fusion layer to obtain the final score of the query image belonging to each category.

[0012] Step (5): The category corresponding to the maximum score is the predicted category. The cross-entropy loss function is calculated with the category of the real sample image corresponding to the query image, and the gradient is backpropagated to update the model parameters.

[0013] Furthermore, in step (1), the support set and query set share the same label space. Unlike general classification tasks, both the support set and query set are used in the training phase. If the support set S contains N classes, and each class contains K (e.g., 1 or 5) samples, this small-shot classification task is called N-way K-shot. It should be noted that the support set has only a small number of samples in each class, which is insufficient to train an effective deep learning network for general classification tasks. Therefore, an auxiliary set is usually introduced for use in the testing phase to learn transferable knowledge. The biggest difference from general classification tasks is that the auxiliary set usually has more classes than the support set and its label space does not overlap with the support set.

[0014] Step (2): The feature extraction network uses Conv-64F, which contains four convolutional blocks. Each convolutional block includes Conv2d, BatchNorm2d, ReLU, and MaxPool2d processing steps. This feature extraction network extracts rich local features F from RGB images. ω The specific formula is expressed as follows:

[0015]

[0016] Among them, F ω It is a three-dimensional feature matrix, where ω represents the network parameters. Let F... ω (X) can be viewed as a feature matrix composed of c×n local features, x i Let represent the i-th local feature in the feature matrix, c represent the number of feature channels, and n represent the number of local features, i.e., n = w × h, where w and h represent the width and height of the feature matrix, respectively. For each support class, all images in that class are mapped to . The spatial representation is used to depict the distribution of the original features of this class. For each query image, n local features are used to estimate the query image's position in the space. Distribution in space.

[0017] In step (3), for selecting the metric branch of local relations, this invention designs a local relation selection module. Local relations refer to the query image's local relation selection method. Each feature vector in the space corresponds to a certain class in the query set. The cosine similarity among all local features in the support space is taken as the maximum value, which represents the cosine similarity between that local feature and the support class. Furthermore, a similar operation is performed on all local features of the query image to obtain the original cosine similarity score I, indicating whether the query image belongs to that class. The specific formula is as follows:

[0018]

[0019] in, s represents all samples of a certain class in the support set, K1 represents the number of samples of that class, and q represents the query image to be predicted. TopK(.) selects the K highest similarity scores from all samples s of a certain class as the similarity between a local feature of q and that class at a certain location, and local represents the local relation selection metric branch. The sum of the similarity scores calculated for n local features in the predicted query image q is... It is the score of the query image q to be predicted belonging to category s.

[0020] The n local features of the query image q to be predicted cannot fully represent the foreground features of the image, and may be subject to interference from background features. Therefore, this invention designs a local feature relationship selection module to perform a weighted summation of the similarity of these n local features, adaptively selecting the important relationships corresponding to the foreground features and suppressing the trivial relationships corresponding to the background features, rather than the simple summation in Formula 2.

[0021] The input to the local feature relation selection module is the feature pair F obtained by the feature extraction module. q F s The output is the similarity weight graph A corresponding to the query image q to be predicted. q First, a 1x1 convolution kernel is used to apply F to the feature pairs. q F s Dimensionality reduction is performed, that is, the number of channels c is transformed into c'. After dimensionality reduction, the feature pairs become Next, the feature pairs after dimensionality reduction are calculated. cosine similarity The formula is as follows:

[0022]

[0023] Where x represents a local feature of the feature map, x q Represents the local features of the query image, x s This represents the local features of a certain class of images in the support set; sim(.,.) represents the cosine similarity between two local features. Then, the cosine similarity... The input is fed into a module consisting of two layers of four-dimensional convolutions to obtain an optimized similarity tensor. The goal is to eliminate unreliable similarity and optimize the similarity tensor. Calculate similarity weights in graph A q A q This reveals the weights of various local features of the query image in measuring the probability that the query image belongs to a certain class. The specific formula is expressed as follows:

[0024]

[0025] Where, x q This represents the local features of the query image, where γ represents the temperature coefficient. This represents the local features of the query image relative to the local features x of a certain class of sample images in the support set. s The similarity is calculated by assigning importance, i.e., a similarity weighted graph. Next, the original cosine similarity scores I (Formula 2) calculated between the query image and a certain class of sample images in the support set are weighted and summed. The specific formula is as follows:

[0026]

[0027] Among them, Sim in Formula 5 local This indicates that the score of the query image belonging to a certain category is obtained based on local relations.

[0028] The global feature distribution metric branch described in step (3) will be explained in detail below.

[0029] The main function of the global feature distribution measurement branch is to obtain the feature distribution statistics of the query image and the support set based on local feature statistics, and to measure the difference between the feature distribution of the query image and a certain support class. This invention employs a convolutional attention mechanism module to sequentially perform channel-level attention and spatial-level attention on the input feature map. Specifically, given the feature pairs F extracted by the feature extractor... q F s The channel-level attention module obtains the weight relationships of each channel. The channel-level attention module first performs space compression using global average pooling and global max pooling to obtain two weight tensors. The two weight tensors are then merged and passed through a multilayer perceptron (MLP) to obtain the channel attention weights W. c The specific formula is expressed as follows:

[0030]

[0031] In Formula 6, ω c The parameters of the multilayer perceptron (MLP) are represented by δ, which represents the sigmoid activation function. This is further used to obtain the channel-level attention map. This represents element-wise multiplication, where F represents the input feature F. q F s The Spatial Attention Module (SWA) uses max pooling and average pooling to compress the channels, resulting in two weight maps. Then, the two weight maps are convolved using a 1x1 layer convolution. The spatial attention weight map W is obtained by fusion. s The specific formula is expressed as follows:

[0032]

[0033] Where C represents a 1*1 convolutional layer, ωs This represents the convolutional layer parameters, and Cat represents the connection operation. Next, the final refined feature map is obtained.

[0034] The feature pairs extracted by the feature extractor are F q F s The feature F corresponding to the image will be queried. q Support set corresponding to a certain type of feature F s The input is fed into the convolutional attention mechanism module to extract features at different scales. This invention uses 21*21, 15*15, 10*10, and 7*7 as the output feature scale set L. For a given scale, the query image features F are calculated. q and support set of a certain type of feature F s The distribution statistics. Assuming it follows a Gaussian distribution, the distribution of the queried image features follows... The feature distribution of a certain class of support sets follows in Let represent the mean vector and covariance matrix, respectively. The similarity between the query image and a certain class of the support set at a specific scale is calculated using KL divergence. The similarity of feature distributions at all scales is then summed to obtain a multi-scale feature distribution metric. The specific formula is as follows:

[0035]

[0036]

[0037] Among them, in formula 8 Indicates F at scale i q With F s The KL divergence, i∈L. Sim global This represents the sum of distributional similarities between the query image and a certain class in the support set across all scales.

[0038] For step (4), based on the similarity scores obtained from the two strategies above, this invention designs an adaptive fusion module to adaptively weight and sum the scores obtained from the two strategies. This fusion module uses a learnable two-dimensional weight vector [ω1, ω2] to autonomously weight the two scores, as expressed in the following formula:

[0039] Sim = ω1.Sim local +ω2.Simg lobal (Formula 9)

[0040] In Formula 9, ω1 and ω2 are learnable parameters, and Sim represents the final prediction result of the query image obtained by adaptively fusing the prediction results of the two strategies.

[0041] The beneficial effects of this invention are as follows:

[0042] This invention proposes a novel few-shot metric learning method that uses local features for metric learning. Compared to previous instance-level metrics, local feature metrics offer richer representations. Based on this, this invention designs two specific metric methods: feature relation selection metric and feature distribution metric. The feature relation selection metric uses cosine similarity to calculate the sum of the optimal similarities between all query local features and a certain class of local features in the support set to measure the similarity between the query image and the support class. Not every similarity score of a query feature is important; foreground features need to be strengthened, while background features need to be weakened. Therefore, this invention designs a local relation selection module to enhance the scores corresponding to foreground features and suppress the scores corresponding to background features. The local relation selection strategy focuses on local information, while the feature distribution metric strategy focuses on the global feature distribution information of the query image and the support class. The feature distribution metric calculates the distribution statistics of query features and support class features, using KL divergence to measure the distribution difference between the query image and a certain support class image, thus deriving the predicted score of the query image. Single-scale distribution cannot effectively measure feature distribution; therefore, this invention integrates a convolutional attention mechanism (CBAM) module based on the feature distribution metric strategy. The aim is to extract multi-scale features from the original features, enhance the semantic information of the original feature space, and make the feature distribution more representative. To adaptively fuse the two metric scores, this invention designs an adaptive fusion module to adaptively weight and fuse the results of the two strategies to obtain the final similarity score. Experimental results on public datasets show that this invention outperforms existing methods. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the specific process of the present invention.

[0044] Figure 2 This is a schematic diagram of the network framework of the present invention.

[0045] Figure 3 This is a schematic diagram of the local relationship selection module of the present invention. Detailed Implementation

[0046] The present invention will be further described in detail below with reference to the accompanying drawings:

[0047] Figure 1 This is a specific flowchart of the present invention. Figure 2 The overall model framework diagram of the present invention is shown. Figure 3 The diagram shows the structure of the local relationship selection module in the model framework diagram.

[0048] like Figure 1As shown, this invention provides a small-sample image classification method that integrates relation selection and multi-scale distribution. This invention solves the problem that instance-level metric methods cannot fully utilize the rich information of local features. Specific process details are attached. Figure 2 As shown. Figure 2 The model structure of this invention is illustrated. The feature extractor extracts rich local features. The local relation selection metric branch first calculates the cosine similarity of each position of the query feature, and then uses the attention map generated by the local relation selection module to strengthen foreground feature relationships and suppress background feature relationships. The multi-scale feature distribution metric branch is based on KL divergence and uses multi-scale feature distribution information to measure the overall distribution difference between the query image and the supporting classes. Finally, the results generated under the two strategies are input into the adaptive fusion module to obtain the class score of the query image. Figure 3 The diagram illustrates the structure of the local relation selection module designed in this invention. First, the cosine similarity of local features is calculated, resulting in a four-dimensional matrix C, which is then input into module h. Module h consists of two four-dimensional convolutional layers, subsequently yielding four-dimensional features. The relation weight graph A is obtained using formula 4.

[0049] The experiments of this invention were conducted on MiniImageNet to verify the effectiveness of the proposed model from three aspects: the effectiveness of the local feature relation selection branch, the multi-scale feature distribution measurement branch, and the joint measurement of the two strategies. First, to verify the effectiveness of the local relation selection module, a comparative experiment was conducted with and without the relation selection module. The experimental results are shown in Table 1.

[0050] Table 1. Impact of Local Feature Relationship Selection Branch and Relationship Selection Module on Experiment

[0051]

[0052]

[0053] Table 1 shows the average accuracy of 5way-1shot and 5way-5shot tasks on the MiniImageNet dataset, with a 95% confidence interval. The experimental results show that using the local relation selection module achieves a higher average accuracy than using only cosine similarity, illustrating the inappropriateness of simply adding the similarities of local features in the query image and demonstrating that the local relation selection module can adaptively weight the similarities of various local features in the query image. Next, this invention verifies the effectiveness of the multi-scale feature distribution measurement strategy; the experimental results are shown in Table 2.

[0054] Table 2 Experimental Results of Multiscale Feature Distribution Measurement

[0055] KL divergence 53.34±0.66 71.45±0.5 Global feature distribution measurement branch 54.22±0.66 71.48±0.51

[0056] Regarding the selection of the output scale of the Convolutional Attention Module (CBAM) under the Global Feature Distribution Measurement branch, this invention experimentally determined that 21*21, 15*15, 10*10, and 7*7 are the optimal choices. The original features are extracted at different scales under the CBAM convolutional module. At each scale, the distribution statistics of the query image and supporting classes are calculated, and the KL divergence is also calculated. The KL divergence measures the difference between the query image and a specific supporting class from the perspective of global feature distribution. Experimental results show that the multi-scale feature distribution measurement method has a higher average accuracy than the single-scale feature distribution measurement method. The main reason is that considering the contextual information of features at multiple scales allows the model to better understand the image content, resulting in more representative distribution statistics.

[0057] Finally, the results of fusing the two strategies using the adaptive fusion module were compared with the Baseline results, as shown in Table 3.

[0058] Table 3. Experimental results of two-branch fusion

[0059] Cosine + KL (Baseline) 54.17±0.63 72.45±0.51 Ours 55.41±0.62 73.26±0.52

[0060] The results show that the final method proposed in this invention has a higher average accuracy than Baeline, demonstrating the effectiveness of the small-sample image classification method proposed in this invention, which combines fusion relationship selection and multi-scale feature distribution measurement.

Claims

1. A small-sample image classification method that integrates relation selection and multi-scale distribution, characterized in that... Includes the following steps: Step (1) Dataset acquisition and creation: The MiniImageNet public dataset is used. In the few-shot learning scenario, the dataset is divided into three sets: the support set and query set for the training phase, and the auxiliary set for the testing phase. Step (2): Using Conv-64F as a feature extractor, input the RGB image into the feature extractor to obtain the initial feature pairs F. q ,F s ; Step (3): Pair the initial features with F q ,F s The inputs are respectively fed into the local relation selection metric branch and the global feature distribution metric branch; the local relation selection metric branch uses the relation selection module to autonomously weight and sum the local relations between the query image and the supporting class to obtain the preliminary prediction score I of the query image; the global feature distribution metric branch measures the difference in distribution between the query image and the supporting class based on the statistical information of the local features to obtain the preliminary prediction score II of the query image. Step (4): The preliminary prediction scores I and II of the query image obtained under the two strategies in step (3) are merged and then input into the adaptive fusion layer to obtain the final score of the query image belonging to each category; Step (5): The category corresponding to the maximum score is the predicted category. The cross-entropy loss function is calculated with the category of the real sample image corresponding to the query image, and the gradient is backpropagated to update the model parameters.

2. The method for classifying small-sample images based on fusion relationship selection and multi-scale distribution according to claim 1, characterized in that... In step (1), the support set and the query set share the same label space, and in this method, both the support set and the query set are used in the training phase.

3. The method for classifying small sample images based on fusion relationship selection and multi-scale distribution according to claim 1, characterized in that... In step (2), the feature extraction network uses Conv-64F, which contains four convolutional blocks. Each convolutional block includes Conv2d, BatchNorm2d, ReLU, and MaxPool2d processing steps. This feature extraction network extracts rich local features F from the RGB image. ω The specific formula is expressed as follows: Among them, F ω It is a three-dimensional feature matrix, where ω represents the network parameters; F ω (X) can be viewed as a feature matrix composed of c×n local features, x i Let represent the i-th local feature in the feature matrix, c represent the number of feature channels, and n represent the number of local features, i.e., n = w × h, where w and h represent the width and height of the feature matrix, respectively; for each support class, all images in that class are mapped to . The spatial representation is used to depict the distribution of the original features of this class. For each query image, n local features are used to estimate the query image's position in the space. Distribution in space.

4. A small-sample image classification method based on fusion relationship selection and multi-scale distribution according to claim 1, 2, or 3, characterized in that... The local relation selection metric branch in step (3) includes a local relation selection module; it calculates the original cosine similarity score I of the query image belonging to a certain class in the support set, and the specific formula is as follows: in, s represents all samples of a certain class in the support set, K1 represents the number of samples of that class, and q represents the query image to be predicted; TopK(.) represents selecting the K highest similarity scores from all samples s of a certain class as the similarity between a local feature of q and that class at a certain location, and local represents the local relation selection metric branch; the sum of the similarity scores calculated for n local features in the predicted query image q is, i.e. It is the score of the query image q to be predicted belonging to category s; The n local features of the query image q to be predicted cannot fully represent the foreground features of the image. Therefore, the local feature relationship selection module is used to weight and sum the similarities of these n local features, adaptively select the important relationships corresponding to the foreground features, and suppress the trivial relationships corresponding to the background features. The input to the local feature relation selection module is the feature pair F obtained by the feature extraction module. q F s The output is the similarity weight graph A corresponding to the query image q to be predicted. q First, a 1x1 convolution kernel is used to apply F to the feature pairs. q F s Dimensionality reduction is performed, that is, the number of channels c is transformed into c'. After dimensionality reduction, the feature pairs become Next, the feature pairs after dimensionality reduction are calculated. cosine similarity The formula is as follows: Where x represents a local feature of the feature map, x q Represents the local features of the query image, x s This represents the local features of a certain class of images in the support set; sim(.,.) represents the cosine similarity between two local features; then the cosine similarity is... The input is fed into a module consisting of two layers of four-dimensional convolutions to obtain an optimized similarity tensor. Using the optimized similarity tensor Calculate similarity weights in graph A q A q This reveals the weights of various local features of the query image in measuring the probability that the query image belongs to a certain class. The specific formula is expressed as follows: Where, x q This represents the local features of the query image, where γ represents the temperature coefficient. This represents the local features of the query image relative to the local features x of a certain class of sample images in the support set. s The similarity is calculated by assigning importance to each image, i.e., a similarity weighted graph. Next, the original cosine similarity scores (I) calculated for the query image and a certain class of sample images in the support set are weighted and summed, as shown in the following formula: Among them, Sim in Formula 5 local This indicates that the score of the query image belonging to a certain category is obtained based on local relations.

5. The method for classifying small sample images based on fusion relationship selection and multi-scale distribution according to claim 4, characterized in that... The global feature distribution measurement branch in step (3) uses a convolutional attention mechanism module to sequentially execute the channel-level attention module and the spatial-level attention module on the input feature map; Given the feature pairs F extracted by the feature extractor q F s The channel-level attention module obtains the weight relationships of each channel; the channel attention module first performs space compression through global average pooling and global max pooling to obtain two weight tensors. The two weight tensors are then merged and passed through a multilayer perceptron (MLP) to obtain the channel attention weights W. c The specific formula is expressed as follows: In Formula 6, ω c The parameters of the multilayer perceptron (MLP) are represented by δ, which represents the sigmoid activation function, resulting in a channel-level attention map. This represents element-wise multiplication, where F represents the input feature F. q F s ; The spatial attention module uses max pooling and average pooling to compress the channels, resulting in two weight maps. Then, the two weight maps are convolved using a 1x1 layer convolution. The spatial attention weight map W is obtained by fusion. s The specific formula is expressed as follows: Where C represents a 1*1 convolutional layer, ω s This represents the convolutional layer parameters, and Cat represents the connection operation; then, the final refined feature map is obtained.

6. The method for classifying small sample images based on fusion relationship selection and multi-scale distribution according to claim 5, characterized in that... The feature pairs extracted by the feature extractor are F q F s The feature F corresponding to the image will be queried. q Support set corresponding to a certain type of feature F s The input is fed into the convolutional attention mechanism module to extract features at different scales, using 21*21, 15*15, 10*10, and 7*7 as the output feature scale set L; for a given scale, the query image features F are calculated respectively. q and support set of a certain type of feature F s Distribution statistics; assuming a Gaussian distribution, the distribution of the queried image features follows a Gaussian distribution. The feature distribution of a certain class of support sets follows in Let represent the mean vector and covariance matrix, respectively. The similarity between the query image and a certain class of feature distributions in the support set at a certain scale is calculated using KL divergence. The similarity of feature distributions at all scales is then summed to obtain a multi-scale feature distribution metric. The specific formula is as follows: Among them, in formula 8 Indicates F at scale i q With F s The KL divergence, i∈L; Sim global This represents the sum of distributional similarities between the query image and a certain class in the support set across all scales.

7. The method for classifying small sample images based on fusion relationship selection and multi-scale distribution according to claim 6, characterized in that... Step (4) includes an adaptive fusion module that adaptively weights and sums the preliminary prediction score I of the query image obtained from the local relation selection metric branch and the preliminary prediction score II of the query image obtained from the global feature distribution metric branch. This adaptive fusion module uses a learnable two-dimensional weight vector [ω1, ω2] to autonomously weight the two scores. The specific formula is as follows: Sim = ω1.Sim local +ω2.Sim global (Formula 9) In Formula 9, ω1 and ω2 are learnable parameters, and Sim represents the final prediction result of the query image obtained by adaptively fusing the prediction results of the two strategies.

Images