A small sample fine-grained image classification method based on local feature screening
By constructing support sets and query sets, representative local features are selected, and the problems of data lack and overfitting in small sample image classification are solved, efficient fine-grained image classification is achieved, and the adaptability and accuracy of the model are improved.
Patent Information
- Application Number
- CN202210885422.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-26
- Publication Date
- 2025-09-02
- Estimated Expiration
- 2042-07-26
AI Technical Summary
In the field of computer vision, the prior art is difficult to effectively solve the problem of local feature screening in small sample image classification, which is of low data in small sample image classification, especially in fine-grained image classification tasks, which are prone to overfitting and difficult to adapt to subtle feature differences.
Using a method based on local feature filtering, by constructing support sets and query sets, using embedded networks to generate feature images, compute cosine distance and similarity matrix, introduce mutual selection and alignment modules to filter out representative local features, and combine them with the TOP-k filtering module for final classification.
Improves the performance of small sample image classification, reduces overfitting problems, and performs excellently on challenging images with good portability and high classification accuracy.
Smart Images
Figure CN115049883B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and in particular relates to a method for small sample fine-grained image classification based on local feature screening. Background Art
[0002] Image classification is a fundamental task in computer vision, aiming to classify images into distinct categories with minimal classification error. While deep neural network learning has achieved significant success in image classification in recent years, this is largely due to the completeness of the training data, which requires carefully annotated, low-noise, and sufficiently large datasets. In many practical scenarios requiring specialized labeling knowledge, such as fine-grained image classification of different bird species, obtaining a large number of well-labeled training samples is often difficult.
[0003] In recent years, research on small-shot image classification algorithms has garnered increasing attention. The goal is to train classification models on known categories, leveraging this learned knowledge to achieve good performance on unknown categories with limited labeled data. Since small-shot image classification relies on a small and limited number of training examples, overfitting is inevitable. To address this issue, some researchers have proposed using meta-learning to address data scarcity. Meta-learning methods typically employ a "learning to learn" paradigm to learn a meta-learner. This approach trains a robust, transferable base model that easily generalizes to new training examples. Common meta-learning models include matching networks, prototype networks, and relational networks. Their common feature is the introduction of metric learning. Metric learning primarily focuses on image feature representation and relational metrics: It maps images into an embedding space to learn high-dimensional image representations, and computes distances between images (such as Euclidean and cosine distances) to learn relational mappings between images. However, these methods focus on classifying coarse-grained, general object categories, making them less suitable for fine-grained classification tasks with high local feature variation or subtle feature differences. Fine-grained image classification, also known as subcategory image classification, aims to classify coarse-grained, large categories (e.g., cats and dogs) into more detailed subcategories (e.g., a subcategory of the dog category). However, due to the subtle inter-class differences and large intra-class variability between subcategories, fine-grained image classification is more challenging than standard image classification tasks. Research has found that fine-grained image recognition relies more on mining spatially local discriminative parts of the input image, rather than using deep features from a global average pooling layer as image-level representations. Consequently, a growing number of researchers are opting to use local descriptors instead of global features as the result of image embedding. While this approach has achieved success in small-shot image classification tasks, due to the cluttered background and large intra-class variability of the classification images, useful local features can be overwhelmed and lost in low-data conditions. This forces the model to select more representative deep descriptors rather than brute-force use of all local feature information. We have found that selecting more valuable deep descriptors as image feature representations can effectively improve classification performance. Summary of the Invention
[0004] The purpose of the present invention is to propose a small sample fine-grained image classification method based on local feature screening, which addresses the problems of data scarcity and screening of local features with low correlation within the image in small sample image classification in the field of computer vision.
[0005] The technical solution to achieve the purpose of the present invention is: a method for small sample fine-grained image classification based on local feature screening, comprising the following steps:
[0006] Step 1: Construct support set S and query set Q:
[0007] Randomly select C images of different categories, each category contains K+M images, where K images constitute the support set S and M images serve as the query set Q, C∈[1-5], K∈[1-15], M∈[1-5]. Support set (x j ,y j ) represents the jth image x in the support set S j and image x j The corresponding category label y j Query Set (x i ,y i ) represents the i-th image x in the query set Q i and image x i The corresponding category label y i . j=1,2,3……C×K, i=1,2,3……C×K, go to step 2.
[0008] Step 2: Define the embedded network f θ (x):
[0009] For each image x in the support set S j And each image x in the query set Q i Incoming embedding network f θ (x) corresponds to the feature image of the generated support set. and the feature image of the query set Go to step 3.
[0010] Step 3: Calculate the feature images in the query set The cosine distance to the image belonging to category c in the support set is used to obtain the relationship matrix R of category c. c , c∈[0,1,2…,C-1], go to step 4.
[0011] Step 4: Repeat step 3, traverse all categories, and obtain C relationship matrices R c . The relationship matrix R of these C categories c Connect them to get the similarity matrix R, where R represents the feature image in the query set To the semantic similarity of the entire support set S. Go to step 5.
[0012] Step 5: Introduce the mutual selection module and obtain a query image descriptor sequence q related to a specific task based on the similarity matrix R * . Go to step 6.
[0013] Step 6: Introduce the alignment module to align the feature images of the support set Feature images to the query set Align, and get the aligned support set consisting of support images
[0014] According to the query image descriptor sequence q * and the relationship matrix R of category c c , get the attention map A c . Using A c To align each image of the support set category c, until all categories of the support set are traversed, and then all images of all categories of the support set are aligned, and go to step 7.
[0015] Step 7: Introduce the TOP-k screening module for the feature images of the query set and the aligned support set consisting of support images The local features of the query set feature image are obtained through the TOP-k screening module and local features of the aligned support set Go to step 8.
[0016] Step 8: Based on the local features of the query set feature image and local features of the aligned support set Calculate the similarity score of each category, and according to the highest score, obtain the final classification category of the query set image to achieve classification.
[0017] Compared with the prior art, the present invention has the following significant advantages:
[0018] 1) High classification performance. We experimented with our method on three datasets and achieved higher results compared with existing advanced methods.
[0019] 2) It significantly solves the overfitting problem of small sample image classification and achieves better results on more challenging images.
[0020] 3) The present invention is portable and can be easily transplanted to other neural networks without changing the network structure. It is simple to implement and has fewer parameters. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 It is a flow chart of the method of the present invention.
[0022] Figure 2This is a flowchart for implementing classification using the present invention. Images in the support and query sets are embedded in a network to obtain high-dimensional feature representations. The relationship matrix between the support set, query, and image is calculated by calculating cosine distance. A mutual selection module is used to obtain a sequence of query set descriptors with specific tasks. Together, these modules generate an attention graph to align the support set. The top-k deep descriptors of the support and query sets are selected for final classification. DETAILED DESCRIPTION
[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0024] First, let's define the concept of small-shot learning training mechanism: Introduction of support set and query set: The so-called N-way K-Shot training mechanism refers to a support set S containing N different image classes and K labeled examples for each class, and a query set Q, which shares the same label space with S. Small-shot learning aims to complete the classification of unlabeled query set samples using a small number of support set images. Due to the lack of data, it is easy to cause overfitting of classification based solely on the support set S. Therefore, an auxiliary set A is introduced during training to learn transferable knowledge. Since A has a disjoint class label space relative to the support set S, we can obtain a strong model that can adapt well to new tasks.
[0025] The present invention will be further described in detail below with reference to the accompanying drawings:
[0026] Combine Figure 1 and Figure 2 The method for small sample fine-grained image classification based on local feature screening described in the present invention comprises the following specific steps:
[0027] Step 1. Construct support set S and query set Q:
[0028] Randomly extract C images of different categories, each category contains K+M images, where K images constitute the support set S and M images serve as the query set Q, C∈[1-5], K∈[1-15], M∈[1-5]; support set (x j ,y j ) represents the jth image x in the support set S j and image x j The corresponding category label y j Query set (xi ,y i ) represents the i-th image x in the query set Q i and image x i The corresponding category label y i ; j = 1, 2, 3 ... C × K, i = 1, 2, 3 ... C × K. Go to step 2.
[0029] Step 2: Define the embedded network f θ (x), each image x in the support set j and each image x in the query set i , passed into the embedding network f θ (x), generate the feature images of the support set respectively and the feature image of the query set The spatial size of each feature image is h×w×d, as follows:
[0030] The d-dimensional vector at each pixel in the feature image is called a depth descriptor. Each feature image contains h×w depth descriptors, and the length of each descriptor is d. For K images of category c in the support set, the set of depth descriptors s c for:
[0031]
[0032] For each image in the query set, the set of depth descriptors q is:
[0033]
[0034] in, represents the u-th depth descriptor in all support set feature images belonging to category c; q v represents the vth depth descriptor in the query set feature image. Go to step 3.
[0035] Step 3: Calculate the feature images in the query set To the feature image of category c in the support set The cosine distance of category c is obtained to obtain the relationship matrix R c ; Relationship matrix R c And its corresponding position The calculation results are as follows:
[0036]
[0037]
[0038] Go to step 4.
[0039] Step 4: Repeat step 3, traverse all categories, and obtain C relationship matrices R c ; The relationship matrix R of these C categories c Connect them to get the similarity matrix R, where R represents the feature image in the query set to the semantic similarity of the entire support set S, where the similarity matrix R contains hw rows and CKhw columns; go to step 5.
[0040] In step 5, the mutual selection module is introduced to obtain a query image feature sequence q related to a specific task according to the similarity matrix R. * , the mutual selection operations are as follows:
[0041] Search the vth row in the similarity matrix R, that is, the vth depth descriptor in the query set feature image, find the index col_index of the column where the maximum value is located, and at the same time, search the index row_index of the row where the maximum value of the column is located; if the index row_index = v, it means that the depth descriptor q represented by the vth row of the query set image is v It is a task-related depth descriptor that needs to be retained; otherwise, it is meaningless and needs to be eliminated. The specific formula is as follows:
[0042] col_index=argmax u R
[0043] row_index=argmax v R
[0044] Among them, argmax u and argmax v Respectively represent the index of the column and row where the maximum value is found.
[0045] Traverse all rows of the similarity matrix R, that is, v∈[1,2,…,hw], and obtain a query image feature sequence q related to a specific task * . Go to step 6.
[0046] Step 6: Set the feature image of the support set Feature images to the query set Align, and get the aligned support set consisting of support images The specific steps are as follows:
[0047] Step 6-1: Obtain the query image feature sequence q related to the specific task according to the mutual selection module in step 5 * , obtain the mask sequence m=[m1,...,m hw ]:
[0048]
[0049] Step 6-2, through the relationship matrix R of the mask sequence m and category c c , get the attention map A c :
[0050] A c =m×R c
[0051] Step 6-3, in A c Perform SoftMax(·) on the image, normalize the weight of each row to 1, and use it to align each image of the support set category c.
[0052]
[0053]
[0054] in, Denotes the alignment matrix of support set category c. Go to step 7.
[0055] Step 7: Introduce the TOP-k screening module for the query set feature image and the aligned support set consisting of support images The local features of the query set feature image are obtained through the TOP-k screening module and local features of the aligned support set The details are as follows:
[0056] Step 7-1: For the query set feature image The spatial size of each feature image is h×w×d, based on each pixel in the feature image. Accumulate the values of all channels at this position, denoted as p a,b , where a∈[1,2,…,h] and b∈[1,2,…,w], is calculated as follows:
[0057]
[0058] n represents the number of channels.
[0059] Step 7-2, get the p a,b Sort from large to small, select the kth value after sorting, as the threshold δ, retain the features at positions greater than δ, and eliminate the remaining positions to obtain the local features of the query set feature image Among them, k∈[12-16]; for the query set feature image Each position The calculation formula for the screening process is:
[0060]
[0061] Traverse the query set feature image and align the support set feature image to obtain the local features of the query set feature image and local features of the aligned support set Go to step 8.
[0062] Step 8: Based on the local features of the query set feature image and local features of the aligned support set Calculate the similarity score of each category, and according to the highest score, obtain the final classification category of the query set image to achieve classification; the specific process is as follows:
[0063] Calculate local features of the query set feature image and the local features of the aligned support set of category c The cosine distance of class c is obtained to obtain the enhanced relationship matrix of class c.
[0064]
[0065] choose The maximum value of each row in , and accumulated, finally get the classification score score(c) of the query image belonging to category c:
[0066]
[0067] Traverse all categories and obtain the classification scores belonging to all categories. The category with the highest score is the final classification category.
Claims
1. A method for small sample fine-grained image classification based on local feature screening, characterized in that: The following steps are involved: Step 1. Construct support set S and query set Q: Randomly extract C images of different categories, each category contains K+M images, where K images constitute the support set S and M images serve as the query set Q, C∈[1-5], K∈[1-15], M∈[1-5]; support set (x j ,y j ) represents the jth image x in the support set S j and image x j The corresponding category label y j Query set (x i ,y i ) represents the i-th image x in the query set Q i and image x i The corresponding category label y i ; j = 1, 2, 3 ... C × K, i = 1, 2, 3 ... C × M, go to step 2; Step 2: Define the embedded network f θ (x): For each image x in the support set S j And each image x in the query set Q i Incoming embedding network f θ (x) corresponds to the feature image of the generated support set. and the feature image of the query set Go to step 3; Step 3: Calculate the feature images in the query set The cosine distance to the image belonging to category c in the support set is used to obtain the relationship matrix R of category c. c , c∈[0,1,2…,C-1], go to step 4; Step 4: Repeat step 3, traverse all categories, and obtain C relationship matrices R c ; The relationship matrix R of these C categories c Connect them to get the similarity matrix R, where R represents the feature image in the query set To the semantic similarity of the entire support set S; go to step 5; Step 5: Introduce the mutual selection module and obtain a query image descriptor sequence q related to the task according to the similarity matrix R * ; The mutual selection operations are as follows: Search the vth row in the similarity matrix R, that is, the vth depth descriptor in the query set feature image, find the index col_index of the column where the maximum value is located, and at the same time, search the index row_index of the row where the maximum value of the column is located; if the index row_index = v, it means that the depth descriptor q represented by the vth row of the query set image is v It is a task-related depth descriptor that needs to be retained; otherwise, it is meaningless and needs to be eliminated. The specific formula is as follows: col_index=argmax u R row_index=argmax v R Among them, argmax u and argmax v Respectively represent the index of the column and row where the maximum value is searched; Traverse all rows of the similarity matrix R, that is, v∈[1,2,…,hw], and obtain a query image feature sequence q related to the task * ; Go to step 6; Step 6: Introduce the alignment module to align the feature images of the support set Feature images to the query set Align, and get the aligned support set consisting of support images According to the query image descriptor sequence q * and the relationship matrix R of category c c , get the attention map A c ; Using A c To align each image of the support set category c, until all categories of the support set are traversed, and then all images of all categories of the support set are aligned. The specific steps are as follows: Step 6-1: Obtain the query image feature sequence q related to the task according to the mutual selection module in step 5 * , obtain the mask sequence m=[m1,...,m hw ]: Step 6-2, through the relationship matrix R of the mask sequence m and category c c , get the attention map A c : A c =m×R c Step 6-3, in A c Perform SoftMax(·) on the image, normalize the weight of each row to 1, and use it to align each image of the support set category c. in, represents the alignment matrix of the support set category c; Go to step 7; Step 7: Introduce the TOP-k screening module for the feature images of the query set and the aligned support set consisting of support images The local features of the query set feature image are obtained through the TOP-k screening module and local features of the aligned support set Go to step 8; Step 8: Based on the local features of the query set feature image and local features of the aligned support set Calculate the similarity score of each category, and according to the highest score, obtain the final classification category of the query set image to achieve classification.
2. The method for small sample fine-grained image classification based on local feature screening according to claim 1, characterized in that: In step 2, define the embedding network f θ (x), each image x in the support set j and each image x in the query set i , passed into the embedding network f θ (x), generate the feature images of the support set respectively and the feature image of the query set The spatial size of each feature image is h×w×d, as follows: The d-dimensional vector at each pixel in the feature image is called a depth descriptor. Each feature image contains h×w depth descriptors, and the length of each descriptor is d. For the K images of category c in the support set, the set of depth descriptors s c for: For each image in the query set, the set of depth descriptors q is: in, represents the u-th depth descriptor in all support set feature images belonging to category c; q v represents the vth deep descriptor in the query set feature image.
3. The method for small sample fine-grained image classification based on local feature screening according to claim 2, characterized in that: In step 3, the feature images in the query set are calculated To the feature image of category c in the support set The cosine distance of category c is obtained to obtain the relationship matrix R c ; Relationship matrix R c And its corresponding position The calculation results are as follows:
4. The method for small sample fine-grained image classification based on local feature screening according to claim 3, characterized in that: In step 4, repeat step 3, traverse all categories, and obtain C relationship matrices R c ; The relationship matrix R of these C categories c Connect them to get the similarity matrix R, where R represents the feature image in the query set to the semantic similarity of the entire support set S, where the similarity matrix R contains hw rows and CKhw columns; go to step 5.
5. The method for small sample fine-grained image classification based on local feature screening according to claim 1, characterized in that: In step 7, the TOP-k screening module is introduced to query the feature image and the aligned support set consisting of support images The local features of the query set feature image are obtained through the TOP-k screening module and local features of the aligned support set The details are as follows: Step 7-1: For the query set feature image The spatial size of each feature image is h×w×d, based on each pixel in the feature image. Accumulate the values of all channels at this position, denoted as p a,b , where a∈[1,2,…,h] and b∈[1,2,…,w], is calculated as follows: n represents the number of channels; Step 7-2, get the p a,b Sort from large to small, select the kth value after sorting, as the threshold δ, retain the features at positions greater than δ, and eliminate the remaining positions to obtain the local features of the query set feature image Among them, k∈[12-16]; for the query set feature image Each position The calculation formula for the screening process is: Traverse the query set feature image and align the support set feature image to obtain the local features of the query set feature image and local features of the aligned support set 6. The method for small sample fine-grained image classification based on local feature screening according to claim 5, characterized in that: In step 8, the local features of the query set feature image are and local features of the aligned support set Calculate the similarity score of each category, and obtain the final classification category of the query set image based on the highest score to achieve classification; the specific process is as follows: Calculate local features of the query set feature image and the local features of the aligned support set of category c The cosine distance of class c is obtained to obtain the enhanced relationship matrix of class c. choose The maximum value of each row in , and accumulated, finally get the classification score score(c) of the query image belonging to category c: Traverse all categories and obtain the classification scores belonging to all categories. The category with the highest score is the final classification category.