Fine-grained small sample image classification system and method based on background suppression and foreground alignment

Abstract

This invention discloses a fine-grained, small-sample image classification system and method based on background suppression and foreground alignment. The classification method includes: obtaining an image feature map through a feature extractor; subjecting the image feature map to a background suppression module to obtain a cropped image, and then obtaining a feature map of the cropped image through a feature extractor; aligning the feature map of the support image with the feature map of the query image as a template to eliminate misalignment between the image pairs; and calculating the similarity between the support image and the query image using a local similarity metric. This invention is simple and efficient, addressing both the problem of model overfitting caused by a lack of samples and the difficulty of fine-grained image recognition.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a fine-grained small sample image classification system and method based on background suppression and foreground alignment. Background Art

[0002] Few-shot learning has gained widespread attention in computer vision and multimedia because it mimics the human ability to learn new concepts. Compared to traditional classification paradigms, few-shot learning does not rely on large-scale labeled datasets and can be easily applied to many real-world scenarios with only very sparse training examples. Fine-grained recognition is a popular and challenging problem that aims to identify images belonging to multiple subcategories (e.g., birds, dogs, cars) of a supercategory. Given that manual annotation of fine-grained images requires domain-specific expertise and collecting high-quality, well-labeled, large-scale datasets is laborious and time-consuming, fine-grained recognition is a suitable application scenario for few-shot learning. Distances (e.g., Euclidean and cosine distances) are then directly computed between support images and the query image in the embedded space. A comparator identifies the query image based on the distances to each support image. Both approaches significantly improve the performance of FSL on general datasets, but rarely achieve ideal performance on fine-grained datasets due to high intra-class variance and low inter-class variance.

[0003] To effectively learn from limited data, researchers have explored many general categories of few-shot learning algorithms. Generally speaking, the main few-shot learning methods can be roughly divided into two categories: optimization-based methods and metric learning-based methods. Optimization-based methods typically adopt a "learning to learn" approach to generate robust models. Through this approach, the model can easily generalize to new, unseen tasks with only a small number of training samples. Metric learning-based methods typically consist of two components: a feature embedding module and a comparator. The feature embedding module maps the image into an embedding space. Distances (such as Euclidean distance and cosine distance) are then calculated between support images and the query image in the embedding space. The comparator identifies the query image based on the distances between the query image and each support image (class prototype). Both methods have significantly improved the performance of few-shot learning on general datasets, but rarely achieve ideal performance on fine-grained datasets due to the high intra-class variance and low inter-class variance of fine-grained images.

[0004] To transfer models from general datasets to fine-grained datasets, many current methods use manually annotated bounding boxes to remove the influence of background. Each method achieves significant performance improvements with additional annotations. However, achieving better performance with manually annotated bounding boxes runs counter to the original intention of few-shot learning, which is to free humans from the tedious and tedious task of annotation. Therefore, our goal is to remove cluttered background and improve performance on fine-grained datasets when only image-level labels and a small number of training images are available. Summary of the Invention

[0005] The purpose of the present invention is to improve a fine-grained small-sample image classification system and method based on background suppression and foreground alignment, which removes cluttered background and improves performance on fine-grained datasets when only image-level labels and a small number of training images are available.

[0006] The technical solution for achieving the purpose of the present invention is as follows: In a first aspect, the present invention provides a fine-grained small sample image classification system based on background suppression and foreground alignment, comprising a feature extractor, a background suppression module, a feature alignment module, and a local similarity measurement module;

[0007] The feature map of the image is obtained through the feature extractor; the image feature map is put into the background suppression module to obtain the cropped image, and then the feature map of the cropped image is obtained through the feature extractor; the feature map of the support image is aligned with the feature map of the query image as a template to eliminate the misalignment problem between the image pairs; the similarity between the support image and the query image is calculated using the local similarity measurement module.

[0008] Furthermore, a meta-learning training method is adopted to divide the dataset into many subtasks; the small task consists of two datasets: a support set and a query set, and then all the images in the task are put into the feature extractor to extract features.

[0009] Furthermore, the background suppression module is used to implement:

[0010] Aggregate the feature map along the channel dimension to obtain the activation map;

[0011] Calculate a threshold, set the activation map above the threshold to 1, and set it below the threshold to 0, and get a foreground mask;

[0012] Calculate the maximum connected component in the foreground mask to obtain the coordinates of the minimum bounding box of the foreground object;

[0013] The original image is cropped and enlarged to its original size according to the obtained bounding box coordinates;

[0014] The cropped image is put back into the feature extractor to extract features, and two feature maps are obtained: one for the original image and the other for the cropped image.

[0015] Furthermore, the feature alignment module is used to implement: calculating the similarity between each local feature of the support image and the query image to obtain a relationship matrix; and reconstructing the feature map of the support image based on the value of the relationship matrix as a weight.

[0016] Furthermore, the local feature similarity measurement module is used to implement: removing the maximum pooling and mean pooling operations, retaining the original dimension of the feature map; averaging the feature maps of all supporting images in each class as the prototype of the class; based on the nearest neighbor idea, calculating the similarity between the feature map of the query image and each class prototype, and averaging the similarities; the query image is identified as the class prototype with which it has the highest similarity.

[0017] In a second aspect, the present invention provides a fine-grained small sample image classification method based on background suppression and foreground alignment, comprising:

[0018] Step 1: Use meta-learning training to divide the dataset into many subtasks;

[0019] Step 2: Put the extracted image feature map into the background suppression module to remove the cluttered background in the image, obtain the cropped image and put it into the feature extractor to extract features;

[0020] Step 3: Use the feature alignment module to reconstruct the feature map of the support image using the feature map of the query image as a template;

[0021] Step 4: Use the local feature similarity measurement module to measure the similarity.

[0022] Furthermore, the small task in step 1 consists of two datasets: support set and query set Q = {(x i ,y i )|i=1…|Q|}, and then put all the images in the task into the feature extractor Θ to extract features, that is, where x i ∈S∪Q; where c is the number of channels, h and w represent the size of the feature map, N represents the number of categories in the small task, and K represents the number of labeled support images for each category.

[0023] Furthermore, the specific operation steps of the background suppression module described in step 2 are as follows:

[0024] Step 201: Aggregate the feature map along the channel dimension to obtain the activation map

[0025] Step 202: Calculate a threshold The activation map above the threshold is set to 1, and the activation map below the threshold is set to 0 to obtain a foreground mask.

[0026] Step 203: Calculation The maximum connected component in , obtains the coordinates of the minimum bounding box of the foreground object;

[0027] Step 204: cropping the original image according to the obtained bounding box coordinates and enlarging it to the original size;

[0028] Step 205: The cropped image is put back into the feature extractor to extract features, thereby obtaining two feature maps: one for the original image and the other for the cropped image.

[0029] Furthermore, the specific operation steps of the feature alignment module described in step 3 are as follows:

[0030] Step 301: Calculate the similarity between each local feature of the support image and the query image to obtain the relationship matrix

[0031] Step 302: According to the relationship matrix a s|q The values ​​of are used as weights to reconstruct the feature map of the support image.

[0032] Furthermore, the specific operation steps of the local feature similarity measurement module described in step 4 are as follows:

[0033] Step 401: Remove the maximum pooling and mean pooling operations in the traditional method and retain the original dimension of the feature map;

[0034] Step 402: Calculate the average of the feature maps of all supporting images in each class as the prototype of the class; based on the nearest neighbor concept: calculate the similarity between the feature map of the query image and each class prototype;

[0035] Step 403: In step 402, the similarity of each position, ie, the local feature vector, is calculated, and then the average is calculated; the query image is identified as the category with which the prototype has the highest similarity.

[0036] Compared with the existing technology, the present invention has the following significant advantages: (1) It solves the problem of fine-grained image recognition in a small sample setting, which is more challenging than general image recognition; (2) The established model does not introduce any additional parameters other than the feature extractor and global classifier and can be trained end-to-end; (3) It identifies two key aspects to improve the performance of fine-grained small sample tasks, namely, reducing background interference and aligning foreground responses. Based on this, the present invention develops background activation suppression, foreground object alignment, and local-to-local similarity measurement to achieve the above goals. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Flow chart of the method of the present invention.

[0038] Figure 2 Activate the suppression module for the background.

[0039] Figure 3 Foreground object alignment module.

[0040] Figure 4 It is a local similarity measurement module. DETAILED DESCRIPTION

[0041] The present invention provides a fine-grained small-sample image classification system based on background suppression and foreground alignment, which can be trained in an end-to-end manner. The system mainly consists of four modules: a feature extractor, a background suppression module, a feature alignment module, and a local similarity measurement module.

[0042] First, features are extracted using a feature extractor, and then subsequent matching and positioning learning are performed. The role of the background suppression module is to generate a positioning map of the foreground mask based on the activation map, because the positions with higher values ​​in the activation map are usually the positions of the key areas. Specifically, without adding trainable parameters, the present invention generates the coordinates of the target position by processing the feature map, and the module is supervised by the global classification loss. Using the generated bounding box information, the cluttered background is removed by cropping and enlarging, and a finer scale of the image is further obtained. Unlike traditional methods, the present invention incorporates the refined image obtained by the background suppression module into the model for learning, and also takes into account the original image, which effectively alleviates the data scarcity problem of small sample learning.

[0043] Then, a feature alignment module is introduced to reconstruct support features using query features as templates by calculating the semantic similarity matrix.

[0044] Finally, the present invention includes a local similarity measurement module that calculates the local similarity between the features of a given sample pair.

[0045] Combine Figure 1 The present invention also provides a fine-grained small sample image classification method based on the above system, comprising:

[0046] Step 1: Use meta-learning training to divide the general dataset into many small tasks to simulate the test scenario where there are few samples in the future. Given an "N-way K-shot" fine-grained recognition task, the dataset needs to be divided into the support set and query set Q = {(x i ,y i )|i=1…|Q|}, and then put all the images in the task into the feature extractor Θ to extract features, that is, where x i ∈S∪Q. Here c is the number of channels, h and w represent the size of the feature map, N represents the number of categories in the small task, and K represents the number of labeled support images for each category. In addition, the formula for calculating the support class prototype is: in A sample with label class i∈{1,…,N}.

[0047] Step 2: Put the divided small tasks into the model for training to obtain the feature maps of the support image and the query image.

[0048] Step three, such as Figure 2 As shown, the extracted image feature map is put into the background suppression module to remove the cluttered background in the image, and the cropped image is obtained and put into the feature extractor to extract features.

[0049] Specifically, given a query image x q , the corresponding feature map is generated by the feature extractor We know that each channel of the feature map can be regarded as an object detector, denoted as f i (i=1,…,c). If a spatial location in the feature map has a high activation value on most channels, then it is likely to correspond to a key area. Therefore, the feature map F is aggregated along the channel dimension as Then, we activate the map A F Introducing an adaptive threshold To determine which position is part of the critical area. Finally, by comparing A F Each element on Generate a foreground mask Specifically, for a specific position (i, j), if A F The activation value of (i,j) is greater than the threshold, then the corresponding Set to 1, otherwise 0. In formula,

[0050]

[0051] Then, calculate The maximum connected area of ​​​​the foreground is predicted. Based on the position coordinates, in the original image x q The foreground is cropped out and enlarged to the same size as the original image, denoted as Then put it into the feature extractor to extract features.

[0052] Step 4: Using the feature map of the query image as a template, reconstruct the feature map of the support image, such as Figure 3 .

[0053] Specifically, support features are compared locally and query features Previously, it was necessary to first solve the semantic misalignment problem caused by intra-class differences (e.g., changes in target pose and position) and cluttered background. The specific solution is to first spatially transform a pair of features into Convert to Right now and where s i and q i Represents the i-th local descriptor in the feature map. Then, using F s and F q Calculate semantic correlation matrix In addition, we normalize the semantic correlation matrix so that the sum of each row of the matrix is ​​1 after normalization. The formula is as follows:

[0054]

[0055] The matrix calculated by formula (2) is used to support the feature F s and query feature F q Alignment. Alignment feature F s|q By alignment function Get. That is

[0056] Step 5: Remove the general pooling operation and use local features to measure similarity.

[0057] After alignment, the semantic features of the image are no longer affected by position deviation. Figure 4 As shown, given a pair of aligned semantic features F s|q , The total spatial similarity L2L(F s|q ,F q ) can be calculated by the following formula:

[0058]

[0059] where cos(·,·) represents the cosine distance function.

[0060] Finally, the overall loss function is constructed from two parts: one is the global classification loss (general cross entropy loss), which only works during the training phase; the other is the local similarity loss, which is composed of the original image and the refined image. In summary, the loss function is as follows:

[0061]

[0062] Among them, λ is the weight factor.

[0063] As shown in Table 1, the method of the present invention is compared with the existing method:

[0064] Table 1

[0065]

[0066] The proposed method achieves the best performance on the bird dataset CUB-200-2011, the car dataset StanfordCars, and the dog dataset StanfordDogs under 5-way 1-shot and 5-way-shot settings.

Claims

1. A fine-grained small-sample image classification system based on background suppression and foreground alignment, characterized by: It includes feature extractor, background suppression module, feature alignment module and local similarity measurement module; The feature map of the image is obtained through the feature extractor; the image feature map is put into the background suppression module to obtain the cropped image, and then the feature map of the cropped image is obtained through the feature extractor; the feature map of the support image is aligned with the feature map of the query image as a template to eliminate the misalignment problem between the image pairs; the similarity between the support image and the query image is calculated using the local similarity measurement module; Using meta-learning training, the dataset is divided into many subtasks; The small task consists of two datasets Composition: support set and query set, then put all images in the task into the feature extractor to extract features; The background suppression module is used to implement: Aggregate the feature map along the channel dimension to obtain the activation map; Calculate a threshold, set the activation map above the threshold to 1, and set it below the threshold to 0, and get a foreground mask; Calculate the maximum connected component in the foreground mask to obtain the coordinates of the minimum bounding box of the foreground object; The original image is cropped and enlarged to its original size according to the obtained bounding box coordinates; The cropped image is put back into the feature extractor to extract features, and two feature maps are obtained: one for the original image and the other for the cropped image; The feature alignment module is used to: calculate the similarity between each local feature of the support image and the query image to obtain a relationship matrix; and reconstruct the feature map of the support image using the values ​​of the relationship matrix as weights; The local similarity measurement module is used to achieve the following: removing the maximum pooling and mean pooling operations to retain the original dimension of the feature map; averaging the feature maps of all supporting images in each class as the prototype of that class; calculating the similarity between the feature map of the query image and each class prototype based on the nearest neighbor idea and averaging the similarities; and identifying the query image as belonging to the class prototype with which it has the highest similarity.

2. A fine-grained small sample image classification method based on background suppression and foreground alignment, characterized in that: The following steps are involved: Step 1: Use meta-learning training to divide the dataset into many subtasks; The small task consists of two datasets: support set and query set Q = {(x i ,y i )|i=1…|Q|}, and then put all the images in the task into the feature extractor Θ to extract features, that is, where x i ∈S∪Q; where c is the number of channels, h and w represent the size of the feature map, N represents the number of categories in the small task, and K represents the number of labeled support images for each category; Step 2: Put the extracted image feature map into the background suppression module to remove the cluttered background in the image, obtain the cropped image and put it into the feature extractor to extract features. The specific steps are as follows: Step 201: Aggregate the feature map along the channel dimension to obtain the activation map Step 202: Calculate a threshold The activation map above the threshold is set to 1, and the activation map below the threshold is set to 0 to obtain a foreground mask. Step 203: Calculation The maximum connected component in , obtains the coordinates of the minimum bounding box of the foreground object; Step 204: cropping the original image according to the obtained bounding box coordinates and enlarging it to the original size; Step 205: The cropped image is put back into the feature extractor to extract features, and two feature maps are obtained: one for the original image and the other for the cropped image; Step 3: Use the feature alignment module to reconstruct the feature map of the support image using the feature map of the query image as a template. The specific steps are as follows: Step 301: Calculate the similarity between each local feature of the support image and the query image to obtain the relationship matrix Step 302: According to the relationship matrix a s|q The value of is used as the weight to reconstruct the feature map of the support image; Step 4: Use the local similarity measurement module to measure the similarity. The specific steps are as follows: Step 401: Remove the maximum pooling and mean pooling operations and retain the original dimension of the feature map; Step 402: Calculate the average of the feature maps of all supporting images in each class as the prototype of the class; calculate the similarity between the feature map of the query image and each class prototype based on the nearest neighbor concept; Step 403: Step 402 calculates the similarity of each position, that is, the local feature vector, and then calculates the average; the query image is identified as the category with which the prototype has the highest similarity.

Images