Small sample fine-grained image classification method with adaptive double-layer feature reconstruction enhancement

Abstract

The application discloses a kind of self-adapting double-layer feature reconstruction enhancement small sample fine-grained image classification method, comprising: 1) extracting deep convolutional image features;2) by double-layer of multi-attention mechanism, self-reconstruction is carried out;3) two layers of self-reconstruction features are used to bidirectional reconstruction, and support sample is reconstructed to query sample, and support sample is reconstructed based on query sample;4) according to the various test scenarios designed, the branch weight of four double-layer bidirectional reconstruction is dynamically adjusted;5) the distance between given query sample and its reconstructed corresponding sample is calculated, and the distance between support sample and its respective reconstruction is then weighted sum is carried out to the four distances using the obtained branch weight;6) the classification of query sample is carried out.The method of the application effectively solves the problems of insufficient fitting capacity and insufficient extraction of discriminative local information when using limited query set features to reconstruct a large number of support set features in the prior art.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of small sample image classification, and particularly relates to a small sample fine-grained image classification method based on adaptive double-layer feature reconstruction enhancement. BACKGROUND

[0002] Small sample fine-grained image classification is a difficult problem in the field of computer vision, not only because of the limited number of labeled samples, but also because of the need to distinguish subtle differences to distinguish different sub-categories in fine-grained images. The metric-based method is one of the most commonly used methods to solve this challenge.

[0003] However, the traditional metric-based small sample method has the problems of spatial information loss and local feature misplacement. The recently proposed feature reconstruction-based method effectively solves the problems of spatial information loss and feature misplacement. However, the reconstruction-based method still needs to solve problems such as insufficient fitting ability when reconstructing a large number of support set features using limited query set features and insufficient extraction of discriminative local information. SUMMARY

[0004] The purpose of the present application is to solve the problems of insufficient fitting ability when reconstructing a large number of support set features using limited query set features and insufficient extraction of discriminative local information in the prior art, and to provide a small sample fine-grained image classification method based on adaptive double-layer feature reconstruction enhancement.

[0005] In order to achieve the above-mentioned purpose, the technical scheme adopted by the present application is as follows.

[0006] A small sample fine-grained image classification method based on adaptive double-layer feature reconstruction enhancement comprises the following steps:

[0007] Step S1, extracting deep convolutional image features of each image;

[0008] Step S2, self-reconstructing the deep convolutional image features of each image through double layers of multi-attention mechanism, so that similar local features become more similar and dissimilar local features become more obvious;

[0009] Step S3, bidirectional reconstruction is performed using two layers of self-reconstructed features, support samples are reconstructed based on query samples, and support samples are reconstructed based on query samples;

[0010] Step S4, dynamically adjusting the branch weight of four double-layer bidirectional reconstruction according to various test scenarios designed, and improving the fitting ability of limited feature reconstruction of a large number of features;

[0011] Step S5, calculate the distance between the given query sample and its reconstructed corresponding sample, and the distance between the support samples and their respective reconstructions, and then use the branch weights derived from step S4 to weight sum the four distances;

[0012] Step S6, classify the query sample.

[0013] Specifically, in step S1, the deep convolutional image features of each image are extracted by using two backbone networks Conv-4 and ResNet-12.

[0014] Further, in step S2, the deep convolutional image features of each image are self-reconstructed by a double layer of multi-attention mechanism, and the specific content is as follows:

[0015] First, the feature is reconstructed into r local features on the spatial position, then the sum of the local feature sequence and the corresponding spatial position embedding is calculated as the input, and finally the multi-head attention operation MSA embedded in the Transformer Encoder framework is used to calculate the two-layer output, so that similar local features become more similar and dissimilar local features become more obvious.

[0016] Further, in step S3, the two-layer self-reconstructed features are bidirectionally reconstructed, and the specific content is as follows:

[0017] Given the query feature, reconstruct the support features in a single class, and given the support features in a single class, reconstruct the query feature; for an N-way K-shot classification task, after ELM processing, the first layer and the second layer of the reconstructed support features and the reconstructed query features of the class are obtained, and then they are multiplied by the weights to obtain the results of the first layer and the second layer after bidirectional reconstruction.

[0018] Further, in step S4, the branch weights of the four double-layer bidirectional reconstructions are dynamically adjusted, and the specific content is as follows:

[0019] An adaptive weight adjustment module P is proposed θ for the post-training phase of the model, the adaptive weight adjustment module P θ contains a five-layer fully connected network architecture, similar to the training target of the main training phase, the post-training phase samples multiple N-way K-shot tasks, and obtains two parameters N and K from the randomly sampled tasks, N∈[5,10,15,···,30], K∈[1,5,10,···,30]; and outputs β1, β2, β3, β4, which are used to dynamically adjust the branch weights of the four double-layer bidirectional reconstructions and .

[0020] Further, the calculation of the distance between the given query sample and its reconstructed corresponding sample in step S5, and the distance between the support samples and their respective reconstructions, is as follows:

[0021]

[0022] The four distances are weighted and summed using the branch weights derived from step S4, and the calculation formula is as follows:

[0023]

[0024] In the above formula, λ1, λ2, λ3, λ4 are learnable weight parameters, and their initial values are 0.25, 0.5, 0.5, and 0.8, respectively; τ is a learnable temperature factor, and its initial value is 1.

[0025] Further, the classification of the query sample in step S6 is as follows:

[0026] The test task is provided to the five local optimal models retained during training, and the average of the predicted probabilities is calculated using soft voting to obtain the final classification result.

[0027] Compared with the prior art, the present application has the following beneficial effects:

[0028] The method of the present application uses an adaptive weight adjustment module to dynamically adjust the branch weights of the four double-layer bidirectional reconstructions (two groups in total), and based on the obtained branch weights, the four distances are weighted and summed to classify the query sample, solving the problem of insufficient fitting ability when using a limited number of feature reconstructions to solve a large number of features in the prior art; at the same time, the experimental results across five fine-grained small sample image datasets show that the method of the present application is significantly better than the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0030] Figure 1 is the flowchart of the adaptive double-layer feature reconstruction enhanced small sample fine-grained image classification method of the present application. DETAILED DESCRIPTION

[0031] For the convenience of those skilled in the art to understand and implement the present application, the steps of the method of the present application are described in detail below, and it should be understood that these examples are only used to illustrate the present application and not to limit the scope of the present application. In addition, it should be understood that those skilled in the art can make various modifications or modifications to the present application after reading the content taught by the present application, and these equivalent forms also fall within the scope defined by the claims attached to the present application.

[0032] Embodiments

[0033] The application discloses a kind of self-adapting double-layer feature reconstruction enhancement small sample fine-grained image classification method, comprising the following steps:

[0034] Step S1, extract the deep convolutional image features of each image;

[0035] Step S2, the deep convolutional image features of each image are self-reconstructed by double layers of multi-attention mechanism, so that similar local features become more similar, and dissimilar local features become more obvious;

[0036] Step S3, two layers of self-reconstruction features are used for bidirectional reconstruction, and the query sample is reconstructed using the support sample, and the support sample is reconstructed based on the query sample;

[0037] Step S4, according to the designed various test scenarios, dynamically adjust the branch weight of four double-layer bidirectional reconstruction, improve the fitting ability of limited feature reconstruction of a large number of features;

[0038] Step S5, calculate the distance between the given query sample and its reconstructed corresponding sample, and the distance between the support sample and its respective reconstruction, and then use the branch weight obtained from step S4 to weight sum the four distances;

[0039] Step S6, classify the query sample.

[0040] Specifically, in step S1, two backbone networks Conv-4 and ResNet-12 are used to extract the deep convolutional image features of each image.

[0041] Further, in step S2, the deep convolutional image features of each image are self-reconstructed by double layers of multi-attention mechanism, and the specific content is as follows:

[0042] First, the feature reconstruction is r local features on spatial position, then the sum of the local feature sequence and the corresponding spatial position embedding is calculated as input, and finally the multi-head attention operation MSA embedded in the Transformer Encoder framework is used to calculate the two-layer output, so that similar local features become more similar, and dissimilar local features become more obvious.

[0043] Further, the two-layer self-reconstruction features are used for bidirectional reconstruction in step S3, and the specific content is as follows:

[0044] The support features in a single class are reconstructed given the query features, and the query features are reconstructed given the support features in a single class; for an N-way K-shot classification task, after ELM processing, the first layer and the second layer of the reconstructed support features and the reconstructed query features of the class are obtained, and then they are multiplied by weights to obtain the results of the first layer and the second layer after bidirectional reconstruction.

[0045] Further, the dynamic adjustment of the branch weights of the four double-layer bidirectional reconstruction in step S4 is as follows:

[0046] An adaptive weight adjustment module P is proposed θ for the post-training phase of the model, the adaptive weight adjustment module P θ contains a five-layer fully connected network architecture, similar to the training target of the main training phase, the post-training phase samples multiple N-way K-shot tasks, and obtains two parameters N and K from the randomly sampled tasks, N∈[5,10,15,···,30], K∈[1,5,10,···,30]; and outputs β1, β2, β3, β4, which are used to dynamically adjust the branch weights of the four double-layer bidirectional reconstruction and .

[0047] Further, the distance between the given query sample and its reconstructed corresponding sample, and the distance between the support sample and its respective reconstruction are calculated in step S5, and the calculation formula is as follows:

[0048]

[0049] The four distances are weighted and summed using the branch weights obtained from step S4, and the calculation formula is as follows:

[0050]

[0051] In the above formula, λ1, λ2, λ3, λ4 are learnable weight parameters, and their initial values are 0.25, 0.5, 0.5, and 0.8, respectively; τ is a learnable temperature factor, and its initial value is 1.

[0052] Further, the classification of the query sample in step S6 is as follows:

[0053] The test task is provided to the five local optimal models reserved during training, and the average of the predicted probabilities is calculated using soft voting to obtain the final classification result.

[0054] The superiority of the method of the present application is further illustrated by a specific experiment.

[0055] Experimental setup

[0056] Two backbone networks, Conv-4 and ResNet-12, are used; the Conv4 network consists of four convolutional blocks, each containing 64 3x3 filters with padding of 1 during convolution. In addition, each block contains a BatchNorm, a ReLU activation, and a 2x2 max-pooling operation; given an input image of 3x84x84, the output size after the Conv4 network is 64x5x5; the ResNet-12 consists of four residual blocks, each with three convolutional layers; each convolutional layer is followed by a BatchNorm operation and a leaky ReLU with a negative slope of 0.1; the last convolutional layer in each block employs a 2x2 max-pooling; the input image size is consistent with Conv-4, and the size of the final feature map is 640x5x5.

[0057] For the main training phase, all Conv-4 and ResNet-12 models are trained on three widely used datasets, CUB, Dogs, and Cars, using SGD with Nesterov momentum of 0.9 and a weight decay factor of 5x10-4. A cosine annealing learning rate schedule is adopted, with an initial learning rate of 0.1, decreasing to 0 over 300 epochs, and a total of 1200 epochs. For the Conv-4 model, the 30-way-5-shot scenario is used for training, while for the ResNet-12 model, the 10-way-5-shot scenario is used. In both cases, 15 query images are used for each class. On the two more challenging fine-grained datasets, meta-iNat and tiered meta-iNat, the Conv-4 model is trained using the Adam optimization strategy with a weight decay factor of 5x10-4. The initial learning rate is set to 0.001, halved every 20 epochs, and a total of 100 epochs are trained. During training, the 20-way-5-shot set is used, and for each class, 15 query images are used. In addition to the meta-iNat and tiered meta-iNat directly saving the last epoch model trained, for other datasets, the best-performing model is selected according to the validation set, and validation is performed every 20 epochs.

[0058] For the post-training phase, the best-performing model saved from the main training phase is retrained. Specifically, all other modules are frozen, and only the AWAM module is trained.

[0059] All Conv-4 and ResNet-12 models were trained using SGD with Nesterov momentum of 0.9, while using a weight decay factor of 5x10-4. The initial learning rate was set to 0.01, and for the Conv-4 model, a total of 20 epochs were trained, while for the ResNet-12 model, a total of 5 epochs were trained. In all post-training experiments, the best-performing model was selected according to the validation set, and validation was performed every 1 epoch.

[0060] In all of the above experiments, standard data augmentation techniques were used, including color jittering, random horizontal flipping, and center crop. For all experiments, the average accuracy on 10,000 randomly generated tasks on Dtest was reported under the standard 5-way, 1-shot, and 5-shot settings, with a 95% confidence interval.

[0061] For training the snapshot ensemble version of the model on the CUB, Dogs, and Cars datasets, Conv-4 and ResNet-12 models were trained using a cosine annealing learning rate. The initial learning rate was set to 0.1, and was decreased to 0 every 240 epochs, after which the model was saved. A total of 1200 epochs were trained, resulting in 5 models. The remaining settings were the same as mentioned above for the non-ensemble version. During the evaluation phase, the test tasks were simultaneously provided to the five locally optimal models that were retained during training. Subsequently, the average of the predicted probabilities was calculated using soft voting, resulting in the final classification results comparison, as shown in Table 1 below.

[0062] Table 1. Classification results comparison of the average of the predicted probabilities

[0063]

[0064] The above description is only the preferred embodiment of the present application, and is not intended to limit other forms of the present application. Any skilled person in the art can modify or change the above-mentioned technical content into equivalent embodiments with equivalent changes. However, any simple modification, equivalent change, and modification of the above-mentioned embodiments without departing from the technical solution of the present application, and according to the technical essence of the present application, still belongs to the protection scope of the present application.

Claims

1. A self-adaptive dual-layer feature reconstruction enhanced small sample fine-grained image classification method, characterized in that, The method comprises the following steps: Step S1, extracting deep convolutional image features of each image; Step S2, self-reconstructing the deep convolutional image features of each image through two layers of multi-attention mechanism, so that similar local features become more similar and dissimilar local features become more obvious; Step S3, bidirectional reconstruction is performed on two layers of self-reconstructed features, support samples are reconstructed based on query samples, and query samples are reconstructed based on support samples; Step S4, according to various test scenarios designed, the branch weight of four bidirectional reconstruction branches is dynamically adjusted to improve the fitting ability of limited features to reconstruct a large number of features; Propose an adaptive weight adjustment module P θ For the post-training phase of the model, the adaptive weight adjustment module P θ Contains a five-layer fully connected network architecture, the post-training phase samples multiple N-way K-shot tasks, and obtains two parameters N and K from randomly sampled tasks, N ∈ [5, 10, 15, ···, 30], K ∈ [1, 5, 10, ···, 30]; And output β1, β2, β3, β4, used to dynamically adjust the branch weights of And ; Step S5, calculating the distance between a given query sample and its reconstructed corresponding sample, and the distance between support samples and their respective reconstructions, and then using the branch weight obtained from step S4 to weight and sum the four distances; Step S6, classifying the query sample.

2. The self-adaptive dual-layer feature reconstruction enhanced small sample fine-grained image classification method according to claim 1, characterized in that, In step S1, two backbone networks Conv-4 and ResNet-12 are used to extract deep convolutional image features of each image.

3. The self-adaptive dual-layer feature reconstruction enhanced small sample fine-grained image classification method according to claim 1, characterized in that, In step S2, the deep convolutional image features of each image are self-reconstructed through two layers of multi-attention mechanism, and the specific content is as follows: First, the feature is reconstructed as r local features on the spatial position, then the sum of the local feature sequence and the corresponding spatial position embedding is calculated as the input, and finally the multi-head attention operation MSA embedded in the Transformer Encoder framework is used to calculate the two-layer output, so that similar local features become more similar and dissimilar local features become more obvious.

4. The self-adaptive dual-layer feature reconstruction enhanced small sample fine-grained image classification method according to claim 1, characterized in that, In step S3, bidirectional reconstruction is performed on two layers of self-reconstructed features, and the specific content is as follows: Given the query feature, reconstruct the support feature in a single class, and given the support feature in a single class, reconstruct the query feature; for an N-way K-shot classification task, after ELM processing, the first layer and the second layer of the reconstructed support feature and the reconstructed query feature of the class are obtained, and then they are multiplied by the weight to obtain the result of the first layer and the second layer of bidirectional reconstruction.

5. The self-adaptive dual-layer feature reconstruction enhanced small sample size fine-grained image classification method according to claim 1, characterized in that, In step S5, the distance between a given query sample and its reconstructed corresponding sample, and the distance between support samples and their respective reconstructions are calculated, and the calculation formula is as follows: The branch weight obtained from step S4 is used to weight and sum the four distances, and the calculation formula is as follows: In the above formula, λ1, λ2, λ3, λ4 are learnable weight parameters, and their initial values are 0.25, 0.5, 0.5, and 0.8 respectively; τ is a learnable temperature factor, and its initial value is 1.

6. The self-adaptive dual-layer feature reconstruction enhanced small sample size fine-grained image classification method according to claim 1, characterized in that, In step S6, the query sample is classified, and the specific content is as follows: The test task is provided to the five local optimal models reserved during training, and the average value of the prediction probability is calculated using soft voting to obtain the final classification result.

Images