Self-adaptive double-layer feature reconstruction enhanced small sample fine-grained image classification method

Abstract

The invention discloses a small sample fine-grained image classification method based on adaptive double-layer feature reconstruction enhancement. The method comprises the following steps: 1) extracting deep convolution image features; 2) performing self-reconstruction through double layers of a multi-attention mechanism; 3) performing bidirectional reconstruction by using the two-layer self-reconstruction features, reconstructing a query sample by using the support sample, and reconstructing the support sample based on the query sample; 4) dynamically adjusting the branch weights of the four double-layer bidirectional reconstructions according to various designed test scenes; 5) calculating the distance between a given query sample and a reconstructed corresponding sample, and the distance between a support sample and respective reconstruction, and then carrying out weighted summation on the four distances by using the obtained branch weight; according to the method, the problems that in the prior art, when limited query set features are used for reconstructing a large number of support set features, the fitting capacity is insufficient, and discriminative local information extraction is insufficient are effectively solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of small sample image classification, and in particular relates to a small sample fine-grained image classification method with adaptive double-layer feature reconstruction enhancement. Background Art

[0002] Few-shot fine-grained image classification is a difficult problem in computer vision, not only because of the limited number of labeled samples, but also because of the need to discern subtle differences to distinguish different subcategories in fine-grained images. Metric-based methods are one of the most common approaches to address this challenge.

[0003] However, traditional metric-based small sample methods suffer from the problems of spatial information loss and local feature misalignment. Recently proposed feature reconstruction-based methods effectively solve the problems of spatial information loss and feature misalignment. However, reconstruction-based methods still need to solve problems such as insufficient fitting ability when reconstructing a large number of support set features using limited query set features and insufficient extraction of discriminative local information. Summary of the invention

[0004] The purpose of the present invention is to provide a small sample fine-grained image classification method enhanced by adaptive two-layer feature reconstruction in view of the shortcomings of the prior art, so as to solve the problems of insufficient fitting ability and insufficient extraction of discriminative local information when using limited query set features to reconstruct a large number of support set features in the prior art.

[0005] In order to achieve the above object, the present invention adopts the following technical solution.

[0006] An adaptive two-layer feature reconstruction enhanced small sample fine-grained image classification method comprises the following steps:

[0007] Step S1, extracting deep convolution image features of each image;

[0008] Step S2: The deep convolutional image features of each image are self-reconstructed through the double layers of the multi-attention mechanism, so that similar local features become more similar and dissimilar local features become more obvious;

[0009] Step S3, using two layers of self-reconstruction features to perform bidirectional reconstruction, using support samples to reconstruct query samples, and reconstructing support samples based on query samples;

[0010] Step S4: dynamically adjust the branch weights of the four double-layer bidirectional reconstructions according to various designed test scenarios to improve the fitting ability of limited features to reconstruct a large number of features;

[0011] Step S5, calculating the distance between a given query sample and its reconstructed corresponding sample, and the distance between the supporting samples and their respective reconstructions, and then weighted summing these four distances using the branch weights obtained from step S4;

[0012] Step S6: classify the query samples.

[0013] Specifically, in step S1, the deep convolutional image features of each image are extracted by using two backbone networks Conv-4 and ResNet-12.

[0014] Furthermore, the deep convolutional image features of each image in step S2 are self-reconstructed through the double layers of the multi-attention mechanism, as follows:

[0015] First, the features are reconstructed into r local features at spatial positions, and then the sum of the local feature sequence and the corresponding spatial position embedding is calculated as input. Finally, the multi-head attention operation MSA embedded in the Transformer Encoder framework is used to complexly calculate the two-layer output, thereby making similar local features more similar and dissimilar local features more obvious.

[0016] Furthermore, the bidirectional reconstruction is performed using two layers of self-reconstruction features as described in step S3, and the specific contents are as follows:

[0017] Given the query features, the supporting features in a single class are reconstructed, and given the supporting features in a single class, the query features are reconstructed; for an N-way K-shot classification task, after ELM processing, the first and second layer reconstructed supporting features and the reconstructed query features of the class are obtained, and then they are multiplied by the weights respectively to obtain the results of the first and second layer bidirectional reconstruction.

[0018] Furthermore, the branch weights of the four two-layer bidirectional reconstructions are dynamically adjusted in step S4, and the specific contents are as follows:

[0019] Propose an adaptive weight adjustment module P θ For the post-training stage of the model, the adaptive weight adjustment module P θ It contains a five-layer fully connected network architecture. Similar to the training objectives of the main training phase, the post-training phase samples multiple N-way K-shot tasks, re-obtains two parameters N and K from the randomly sampled tasks, N∈[5,10,15,···,30], K∈[1,5,10,···,30]; and outputs β1, β2, β3, β4 for dynamically adjusting the four two-layer bidirectional reconstruction and The branch weight of .

[0020] Furthermore, the distance between the given query sample and its reconstructed corresponding sample, as well as the distance between the supporting samples and their respective reconstructions, is calculated in step S5 using the following calculation formula:

[0021]

[0022] The branch weights obtained in step S4 are used to perform weighted summation on the four distances, and the calculation formula is as follows:

[0023]

[0024] In the above formula, λ1, λ2, λ3, and λ4 are learnable weight parameters, and their initialization values ​​are 0.25, 0.5, 0.5, and 0.8 respectively; τ is a learnable temperature factor, and its initialization value is 1.

[0025] Furthermore, the query samples are classified in step S6, and the specific contents are as follows:

[0026] The test task is given to the five local optimal models retained during training, and the predicted probabilities are averaged using soft voting to obtain the final classification result.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The method of the present invention designs an adaptive weight adjustment module to dynamically adjust the branch weights of four double-layer bidirectional reconstructions (two groups in total), and performs weighted summation of the four distances based on the obtained branch weights to classify the query samples, which solves the problem of insufficient fitting ability when the prior art uses a limited number of features to reconstruct a large number of features; at the same time, the experimental results across five fine-grained small sample image data sets show that the method of the present invention is significantly better than the existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present disclosure and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 It is a flow chart of the small sample fine-grained image classification method enhanced by adaptive double-layer feature reconstruction of the present invention. DETAILED DESCRIPTION

[0031] In order to facilitate those of ordinary skill in the art to understand and implement the present invention, the following detailed description is given to each step of the method proposed by the present invention, and it should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of the present application.

[0032] Example

[0033] The present invention discloses a small sample fine-grained image classification method with adaptive double-layer feature reconstruction enhancement, comprising the following steps:

[0034] Step S1, extracting deep convolution image features of each image;

[0035] Step S2: The deep convolutional image features of each image are self-reconstructed through the double layers of the multi-attention mechanism, so that similar local features become more similar and dissimilar local features become more obvious;

[0036] Step S3, using two layers of self-reconstruction features to perform bidirectional reconstruction, using support samples to reconstruct query samples, and reconstructing support samples based on query samples;

[0037] Step S4: dynamically adjust the branch weights of the four double-layer bidirectional reconstructions according to various designed test scenarios to improve the fitting ability of limited features to reconstruct a large number of features;

[0038] Step S5, calculating the distance between a given query sample and its reconstructed corresponding sample, and the distance between the supporting samples and their respective reconstructions, and then weighted summing these four distances using the branch weights obtained from step S4;

[0039] Step S6: classify the query samples.

[0040] Specifically, in step S1, the deep convolutional image features of each image are extracted by using two backbone networks Conv-4 and ResNet-12.

[0041] Furthermore, the deep convolutional image features of each image in step S2 are self-reconstructed through the double layers of the multi-attention mechanism, as follows:

[0042] First, the features are reconstructed into r local features at spatial positions, and then the sum of the local feature sequence and the corresponding spatial position embedding is calculated as input. Finally, the multi-head attention operation MSA embedded in the Transformer Encoder framework is used to complexly calculate the two-layer output, thereby making similar local features more similar and dissimilar local features more obvious.

[0043] Furthermore, the bidirectional reconstruction is performed using two layers of self-reconstruction features as described in step S3, and the specific contents are as follows:

[0044] Given the query features, the supporting features in a single class are reconstructed, and given the supporting features in a single class, the query features are reconstructed; for an N-way K-shot classification task, after ELM processing, the first and second layer reconstructed supporting features and the reconstructed query features of the class are obtained, and then they are multiplied by the weights respectively to obtain the results of the first and second layer bidirectional reconstruction.

[0045] Furthermore, the branch weights of the four two-layer bidirectional reconstructions are dynamically adjusted in step S4, and the specific contents are as follows:

[0046] Propose an adaptive weight adjustment module P θ For the post-training stage of the model, the adaptive weight adjustment module P θ It contains a five-layer fully connected network architecture. Similar to the training objectives of the main training phase, the post-training phase samples multiple N-way K-shot tasks, re-obtains two parameters N and K from the randomly sampled tasks, N∈[5,10,15,···,30], K∈[1,5,10,···,30]; and outputs β1, β2, β3, β4 for dynamically adjusting the four two-layer bidirectional reconstruction and The branch weight of .

[0047] Furthermore, the distance between the given query sample and its reconstructed corresponding sample, as well as the distance between the supporting samples and their respective reconstructions, is calculated in step S5 using the following calculation formula:

[0048]

[0049] The branch weights obtained in step S4 are used to perform weighted summation on the four distances, and the calculation formula is as follows:

[0050]

[0051] In the above formula, λ1, λ2, λ3, and λ4 are learnable weight parameters, and their initialization values ​​are 0.25, 0.5, 0.5, and 0.8 respectively; τ is a learnable temperature factor, and its initialization value is 1.

[0052] Furthermore, the query samples are classified in step S6, and the specific contents are as follows:

[0053] The test task is given to the five local optimal models retained during training, and the predicted probabilities are averaged using soft voting to obtain the final classification result.

[0054] The superiority of the method of the present invention is further illustrated by a specific experiment below.

[0055] Experimental setup

[0056] Two backbone networks are used: Conv-4 and ResNet-12; the Conv4 network consists of four convolutional blocks, each of which contains 64 3×3 filters, and the padding is 1 during convolution. In addition, each block contains BatchNorm, ReLU activation, and 2×2 maximum pooling operations; given a 3×84×84 input image, the output size after the Conv4 network is 64×5×5; ResNet-12 consists of four residual blocks, each with three convolutional layers; each convolutional layer is followed by a BatchNorm operation and a leaky ReLU with a negative slope of 0.1; the last convolutional layer in each block uses 2×2 maximum pooling; the input image size is consistent with Conv-4, and the size of the final feature map is 640×5×5.

[0057] For the main training phase, all Conv-4 and ResNet-12 models are trained using SGD with Nesterov momentum of 0.9 and a weight decay factor of 5×10-4 on three widely used datasets - CUB, Dogs and Cars. A cosine annealing learning rate schedule is adopted, where the initial learning rate is set to 0.1 and is reduced to 0 in 300 epochs, and a total of 1200 epochs are trained. For Conv-4 models, 30-way-5-shot episodes are used for training in this example, while for ResNet-12 models, 10-way-5-shot episodes are used. In both cases, 15 query images are used for each class. On two more challenging fine-grained datasets meta-iNat and tieredmeta-iNat, Conv-4 models are trained using the Adam optimization strategy in this example with a weight decay factor of 5×10-4. The initial learning rate is set to 0.001 and is halved every 20 epochs, and a total of 100 epochs are trained. During the training process, a 20-way-5-shot set is used. For each category, 15 query images are used in this example. Except for meta-iNat and tiered meta-iNat, which directly save the last epoch model of training, for other datasets, the best performing model is selected based on the validation set and validated every 20 epochs.

[0058] For the post-training phase, retrain the best performing model saved from the main training phase. Specifically, freeze all other modules and only train the AWAM module.

[0059] All Conv-4 and ResNet-12 models were trained using SGD with Nesterov momentum of 0.9, while using a weight decay factor of 5×10-4. The initial learning rate was set to 0.01, and a total of 20 epochs were trained for the Conv-4 model, while a total of 5 epochs were trained for the ResNet-12 model. In all post-training experiments, the best performing model was selected based on the validation set and validated every 1 epoch.

[0060] In all the above experiments, standard data augmentation techniques are used, including color jittering, random horizontal flipping, and center cropping. For all experiments, the average accuracy of 10,000 randomly generated tasks on Dtest is reported under the standard 5way, 1-shot, and 5-shot settings, with a confidence interval of 95%.

[0061] For the snapshot ensemble version of model training on the CUB, Dogs, and Cars datasets, the Conv-4 and ResNet-12 models were trained using a cosine annealing learning rate. The initial learning rate was set to 0.1 and decreased to 0 every 240 epochs before saving the model. A total of 1200 epochs were trained to obtain 5 models. The rest of the settings were the same as the non-ensemble version mentioned above. During the evaluation phase, this example simultaneously provided the test task to the five local optimal models retained during training. Subsequently, soft voting was used to calculate the average of the predicted probabilities to obtain the final classification result comparison, which is shown in Table 1 below.

[0062] Table 1. Comparison of classification results of average prediction probability

[0063]

[0064] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A small sample fine-grained image classification method with adaptive two-layer feature reconstruction enhancement, characterized in that: The following steps are involved: Step S1, extracting deep convolution image features of each image; Step S2: The deep convolutional image features of each image are self-reconstructed through the double layers of the multi-attention mechanism, so that similar local features become more similar and dissimilar local features become more obvious; Step S3, using two layers of self-reconstruction features to perform bidirectional reconstruction, using support samples to reconstruct query samples, and reconstructing support samples based on query samples; Step S4: dynamically adjust the branch weights of the four double-layer bidirectional reconstructions according to various designed test scenarios to improve the fitting ability of limited features to reconstruct a large number of features; Step S5, calculating the distance between a given query sample and its reconstructed corresponding sample, and the distance between the supporting samples and their respective reconstructions, and then weighted summing these four distances using the branch weights obtained from step S4; Step S6: classify the query samples.

2. According to the adaptive double-layer feature reconstruction and enhanced small sample fine-grained image classification method of claim 1, it is characterized in that: In step S1, deep convolutional image features of each image are extracted by using two backbone networks Conv-4 and ResNet-12.

3. According to the adaptive double-layer feature reconstruction and enhanced small sample fine-grained image classification method of claim 1, it is characterized in that: The deep convolutional image features of each image described in step S2 are self-reconstructed through the double layers of the multi-attention mechanism, as follows: First, the features are reconstructed into r local features at spatial positions, and then the sum of the local feature sequence and the corresponding spatial position embedding is calculated as input. Finally, the multi-head attention operation MSA embedded in the Transformer Encoder framework is used to complexly calculate the two-layer output, thereby making similar local features more similar and dissimilar local features more obvious.

4. According to the method of adaptive double-layer feature reconstruction and enhancement of small sample fine-grained image classification according to claim 1, it is characterized in that: The bidirectional reconstruction is performed using two layers of self-reconstruction features as described in step S3, and the specific contents are as follows: Given the query features, the supporting features in a single class are reconstructed, and given the supporting features in a single class, the query features are reconstructed; for an N-way K-shot classification task, after ELM processing, the first and second layer reconstructed supporting features and the reconstructed query features of the class are obtained, and then they are multiplied by the weights respectively to obtain the results of the first and second layer bidirectional reconstruction.

5. The small sample fine-grained image classification method with adaptive two-layer feature reconstruction enhancement according to claim 1 is characterized in that: The branch weights of the four two-layer bidirectional reconstructions are dynamically adjusted in step S4, and the specific contents are as follows: Propose an adaptive weight adjustment module P θ For the post-training stage of the model, the adaptive weight adjustment module P θ It contains a five-layer fully connected network architecture. Similar to the training objectives of the main training phase, the post-training phase samples multiple N-way K-shot tasks, re-obtains two parameters N and K from the randomly sampled tasks, N∈[5,10,15,···,30], K∈[1,5,10,···,30]; and outputs β1, β2, β3, β4 for dynamically adjusting the four two-layer bidirectional reconstruction and The branch weight of .

6. The small sample fine-grained image classification method with adaptive two-layer feature reconstruction enhancement according to claim 1 is characterized in that: The distance between a given query sample and its reconstructed corresponding sample, as well as the distance between supporting samples and their respective reconstructions, is calculated in step S5 using the following formula: The branch weights obtained in step S4 are used to perform weighted summation on the four distances, and the calculation formula is as follows: In the above formula, λ1, λ2, λ3, and λ4 are learnable weight parameters, and their initialization values ​​are 0.25, 0.5, 0.5, and 0.8 respectively; τ is a learnable temperature factor, and its initialization value is 1.

7. The small sample fine-grained image classification method with adaptive two-layer feature reconstruction enhancement according to claim 1 is characterized in that: The query samples are classified in step S6, and the specific contents are as follows: The test task is given to the five local optimal models retained during training, and the predicted probabilities are averaged using soft voting to obtain the final classification result.

Images