Few-shot fine-grained image recognition method based on component-level filter family

Abstract

The application discloses a few-sample fine-grained image recognition method based on component-level filter family, comprising: extracting feature maps of samples based on a convolutional neural network, and generating global convergence features based on global maximum pooling and global average pooling; transmitting the global convergence features into a meta generator to obtain a component-level filter family, and adding an optimization target in the filter family; performing component-level multiplication on the component-level filter family and the global convergence features to obtain different component-level enhanced features, splicing the different component-level enhanced features to obtain final embedding features, and performing class prediction and model training based on a prototype representation and a nearest neighbor method. The application generates a component-level filter family corresponding to key parts of a fine-grained image through a component-level filter family generator, and makes the generated filter family correspond to different key parts through an optimization target of making the filters as orthogonal as possible, and obtains a component-level enhanced representation conducive to fine-grained image recognition through component-level multiplication on the global convergence features.

Description

Technical Field

[0001] This invention belongs to the field of few-sample fine-grained image recognition, specifically relating to a few-sample fine-grained image recognition method based on a family of component-level filters. Background Technology

[0002] Fine-grained image recognition is a long-standing fundamental research topic in computer vision, aiming to distinguish between subclasses belonging to a specific base class, such as birds, dogs, and cars belonging to different subclasses. This is a challenging task due to the variance between subclasses and within subclasses. With the development of deep learning, many works have achieved good results. However, these results always rely on a large number of labeled samples, or even stronger supervisory information, such as bounding boxes and part information annotations. In contrast, the human visual system can quickly learn new concepts and features from a few examples and then identify similar objects in new images. To mimic this rapid learning ability of humans, few-shot learning has received increasing attention in recent years. Few-shot learning aims to quickly generalize to new tasks containing only a small number of supervised samples after utilizing prior knowledge. Under this setting, recognizing each category requires only a very small number, or even one, labeled sample, thus greatly reducing the cost of manual annotation. This invention studies few-shot fine-grained image recognition, which utilizes prior knowledge to enable the model to perform fine-grained visual recognition tasks well using only a small amount of supervised data.

[0003] Conventional fine-grained image recognition methods struggle to achieve good results in few-shot learning scenarios, primarily because they often require substantial amounts of supervised data to learn new categories. To enable models to effectively recognize fine-grained images with limited supervised samples, researchers proposed a meta-learning framework using bilinear pooling. During training, this meta-learning approach leverages the model's ability to quickly extract knowledge from new samples, rather than simply learning to identify specific categories. Furthermore, bilinear pooling-based models can extract higher-order features from images, which are beneficial for distinguishing subtle differences between fine-grained images. Specifically, the meta-learning framework mimics the method used to construct few-shot test tasks, randomly selecting the same number of categories and samples from the training set (categories disjoint with the test set) to build a meta-training task set. Training on these numerous meta-training tasks imbues the model with meta-learning capabilities. Bilinear pooling involves inputting a fine-grained image into two convolutional neural networks to obtain two different deep activation features. The two local activation features at the same location are then multiplied by an outer product to obtain a local high-order feature matrix. Finally, max pooling is applied to all local high-order feature matrices to obtain a global high-order feature matrix. Leveraging the learning capabilities of meta-learning and the high-order feature extraction capabilities of bilinear pooling, the model possesses a certain performance in solving fine-grained problems with few samples.

[0004] Although the bilinear pooling method based on meta-learning has achieved good classification results, it requires the calculation of high-order features of samples, which has problems such as: 1) high complexity, high computational cost, and complex model training process; 2) the method cannot control the semantic information of the generated new classes well. Summary of the Invention

[0005] The purpose of this invention is to provide a few-sample fine-grained image recognition method based on a family of component-level filters.

[0006] The technical solution to achieve the objective of this invention is as follows: Firstly, this invention provides a few-sample fine-grained image recognition method based on a component-level filter family, comprising the following steps:

[0007] Step 1: Extract feature maps of samples based on convolutional neural networks, and generate global pooled features based on global max pooling and global average pooling;

[0008] Step 2: Input the global pooling features into the meta-generator to obtain the component-level filter family, and add an optimization objective to the filter family to remove redundant information between component-level filters;

[0009] Step 3: Multiply the component-level filter family with the global pooled features to obtain different component-level enhanced features, and then concatenate them to obtain the final embedded features. Class prediction and model training are performed based on prototype representation and nearest neighbor methods.

[0010] In a second aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in the first aspect.

[0011] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0012] Fourthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0013] Compared with the prior art, the significant advantages of this invention are: (1) This method uses a meta-generator to obtain a family of component-level filters corresponding to different key parts of a fine-grained image, which can effectively enhance the representation of key parts of a fine-grained image; (2) The method based on the family of component-level filters can effectively control the feature semantic information of new class samples; (3) Compared with the existing methods based on high-order representation, our method has higher running efficiency. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the few-sample fine-grained image recognition method based on a family of component-level filters according to the present invention. Detailed Implementation

[0015] Combination Figure 1 The present invention provides a few-sample fine-grained image recognition method based on a component-level filter family, comprising the following steps:

[0016] Step 1: Extract feature maps of samples based on convolutional neural networks, and generate global pooled features based on global max pooling and global average pooling;

[0017] For the input sample I i After passing through the convolutional neural network f cnn (·) yields a three-dimensional tensor feature.

[0018] A i =f cnn (I i )∈R H×W×C(1)

[0019] Where H, W, and C represent A respectively. i The feature length, feature width, and number of channels. For A i Two global representation vectors are obtained by applying global max pooling and global average pooling respectively. and And splice them together to obtain global convergence features

[0020]

[0021] GAP(·) is global average pooling, and GMP(·) is global max pooling.

[0022] Step 2: Input the global pooling features into the meta-generator to obtain the component-level filter family, and add an optimization objective to the filter family to remove redundant information between component-level filters;

[0023] Fine-grained image recognition relies on the differentiation of identifiable parts of the sample. Therefore, we propose a component-level filter family generator, which aims to generate a family of filters to enhance the feature representation of identifiable parts in fine-grained recognition.

[0024] The global pooling features obtained in step 1 are fed into the component-level filter family generator to obtain the filter family.

[0025]

[0026] Where Φ(·) is the filter family generator, n is the number of filters generated, and a ij ∈R 2C To ensure the differences between the different filters, we added an optimization objective (4) to these filters to remove redundant information:

[0027]

[0028] Where I is the identity matrix. a is the square of the F-norm. i ∈R 2C×n It is a matrix composed of a family of filters.

[0029] Step 3: Element-wise multiplication of the component-level filter family with the global pooled features yields different component-level enhanced features, which are then concatenated to obtain the final embedded features. Class prediction and model training are then performed based on prototype representation and nearest neighbor methods.

[0030] The global pooled features obtained in step 1 and each filter in the component-level filter family obtained in step 2 are multiplied element-wise to obtain the corresponding component-enhanced representation:

[0031]

[0032] The operator represents prime multiplication. Concatenating all augmented representations yields the final embedding representation of the sample:

[0033] u i =[u i1 ;…;u in ]∈R 2nC (6)

[0034] For all labeled supporting samples, we calculate their embedding features using formulas (1)(2)(3)(5)(6), and then calculate the class prototypes for all categories (taking the calculation of the class prototype for category k as an example):

[0035]

[0036] Where Ω k ={i|y i =k} ​​represents the set of indices of all instances belonging to class k, |Ω k | indicates the size of the index set.

[0037] For any query sample I′, its feature embedding u′ is obtained through the model of this invention, and the predicted distribution of its category y′ is obtained by calculating the softmax of the distance between u′ and all prototypes:

[0038]

[0039] Where d(·) is the distance function. Here we use the square of the Euclidean distance as the distance function.

[0040] The classification loss of the model is obtained by the cross-entropy loss (·):

[0041]

[0042] Where p(y′|u′) is the predicted distribution of the query sample I′. This is the true label of sample I′. The final loss of the model is:

[0043]

[0044] Where λ is the weight hyperparameter of the optimization objective of the component-level filter family generation module.

[0045] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A few-sample fine-grained image recognition method based on a family of component-level filters, characterized in that, Includes the following steps: Step 1: Extract feature maps of samples based on convolutional neural networks, and generate global pooled features based on global max pooling and global average pooling; Step 2 involves inputting the global pooled features into the meta-generator to obtain a family of component-level filters, and adding an optimization objective to the filter family to remove redundant information between component-level filters; specifically: The global pooling features obtained in step 1 are fed into the component-level filter family generator to obtain the filter family. Where Φ(·) is the filter family generator, n is the number of filters generated, and a ij ∈R 2C , This is a global convergence feature; Add optimization objectives between these filters to remove redundant information: Where I is the identity matrix. Let a be the square of the F-norm. i ∈R 2C×n The matrix formed by the filter family; The final loss of the model is: Where λ is the weight hyperparameter of the optimization objective of the component-level filter family generation module. The classification loss of the model; Step 3: Multiply the component-level filter family with the global pooled features to obtain different component-level enhanced features, and then concatenate them to obtain the final embedded features. Class prediction and model training are performed based on prototype representation and nearest neighbor methods.

2. The method for few-sample fine-grained image recognition based on a family of component-level filters according to claim 1, characterized in that, In step 1, for the input sample I i After passing through the convolutional neural network f cnn (·) yields a three-dimensional tensor feature. A i =f cnn (I i )∈R H×W×C Where H, W, and C represent A respectively. i Feature length, feature width, and number of channels; for A i Two global representation vectors are obtained by applying global max pooling and global average pooling respectively. and And splice them together to obtain global convergence features GAP(·) is global average pooling, and GMP(·) is global max pooling.

3. The few-sample fine-grained image recognition method based on a family of component-level filters according to claim 1, characterized in that, In step 3, the component-level filter family is multiplied with the global pooled features to obtain different component-level enhanced features, which are then concatenated to obtain the final embedded features. Class prediction and model training are performed based on prototype representation and nearest neighbor methods. The corresponding component-enhanced representation is obtained by performing a prime product of the global pooled features obtained in step 1 and each filter in the component-level filter family obtained in step 2. The ° operator represents prime product; concatenating all augmented representations yields the final embedded representation of the sample: in i =[in i1 ;…;in in ]∈R 2nC For all labeled supporting samples, first calculate their embedding features. Then calculate the category prototype for all categories: Where Ω k ={i|y i =k} ​​represents the set of indices of all instances belonging to class k, |Ω k | Indicates the size of the index set; For any query sample I ′ The feature embedding u is obtained through the model. ′ By calculating u ′ The softmax of the distance to all prototypes yields its category y. ′ Predicted distribution: Where d(·) is the distance function; The classification loss of the model is obtained by the cross-entropy loss (·): Where p(y) ′ |u ′ ) is the query sample I ′ The predicted distribution It is sample I ′ The true label.

4. The few-sample fine-grained image recognition method based on a family of component-level filters according to claim 3, characterized in that, Use the square of the Euclidean distance as the distance function.

5. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method described in any one of claims 1-4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any of claims 1-4.

7. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1-4.

Images