Adaptive number of discriminative region localization and representation method

Abstract

The application discloses a method for positioning and representing discriminative regions based on adaptive quantity, and mainly comprises the following steps: extracting a deep feature of an input image; performing self-attention transformation on the feature to obtain a self-attention graph, and fusing the self-attention graph with a category activation graph extracted to obtain a saliency graph; extracting a candidate sampling point from the saliency graph, and further expanding the candidate sampling point into discriminative regions with adaptive quantity. The application is used for fine-grained image recognition, can position and represent discriminative regions with adaptive quantity, and can achieve a very high recognition accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a method for locating and representing discriminative regions with an adaptive number of discriminative regions, which is used for fine-grained image recognition and can accurately locate discriminative regions. Background Art

[0002] Unlike general image recognition tasks, fine-grained image recognition tasks require more detailed classification, specifically identifying subcategories within a broad category, such as bird species, aircraft types, and commodity types. Because of the more detailed recognition requirements, fine-grained image recognition has a wider range of applications and demands in real-world scenarios, and has been a key area of ​​focus for both academia and industry.

[0003] Fine-grained image recognition faces three main challenges. First, inter-class differences are minimal; images in one class may differ from those in other classes by only a few subtle object parts. Second, intra-class variance is large; samples of the same class can differ significantly. For example, the different postures of birds and the different growth stages of plants can cause significant variations in the appearance of the samples. Third, there is a shortage of labeled data. Fine-grained image annotation requires advanced domain knowledge, making data annotation extremely difficult and making it difficult to build large-scale datasets to support model training.

[0004] To address these difficulties, existing fine-grained image recognition methods often include a process for localizing discriminative regions, specifically locating key parts of an object. For example, Part R-CNN uses positional annotations of object parts (such as bird heads and claws) to train the localization module and is considered a "strongly supervised" fine-grained image recognition method. However, these annotations are often difficult to obtain. To reduce the need for annotation information, PA-CNN uses a collaborative segmentation approach rather than object part annotations to determine the locations of object parts and is considered a "weakly supervised" fine-grained image recognition method. Many subsequent works also move away from object part annotations, relying solely on category labels. For example, RA-CNN and HS-Net both use classification results to iteratively localize discriminative regions of an object. Another example, NTS-Net, uses three sub-networks: localization, recognition, and fusion, for fine-grained image recognition.

[0005] However, the current discriminative region positioning method still has shortcomings. For different categories to be identified, the number of their discriminative regions is different. For example, for bird A, the discriminative regions are the bird's head and the bird's chest, while for bird B, the discriminative regions are the bird's head, bird's claws and bird's wings. The number of discriminative regions located by the existing discriminative region positioning method is often fixed and determined by pre-set empirical values. This leads to multiple detections and missed detections in practical applications. This will affect the subsequent recognition process and lead to a decrease in recognition effect. Therefore, it is very meaningful to study a discriminative region positioning and representation method with an adaptive number of discriminative regions, which will help improve the efficiency and accuracy of fine-grained image recognition. Summary of the Invention

[0006] To address the aforementioned technical issues in the prior art, this paper proposes an adaptive number of discriminative regions for fine-grained image recognition tasks. Based on deep features extracted by deep neural networks, this paper utilizes a self-attention mechanism and class activation maps to identify candidate extreme points. A trainable Gaussian kernel function is then used to expand these candidate extreme points into discriminative regions. Furthermore, a deep convolutional neural network is used to extract discriminative features, and finally a multi-layer perceptron is trained for fine-grained image recognition.

[0007] The technical solution adopted in the present invention is as follows:

[0008] A method for locating and representing discriminative regions of adaptive quantity, comprising the steps of:

[0009] Step 1. Extract deep features of the input image;

[0010] Step 2. Use the self-attention mechanism on the deep features to obtain a self-attention map to show the correlation between different regions, thereby helping to determine the discriminative regions;

[0011] Step 3. Extract class activation maps for deep features to show the contribution of different regions to each class;

[0012] Step 4. Fuse the self-attention map and the class activation map to obtain candidate extreme points to determine the center position of the discriminative area;

[0013] Step 5. Expand the candidate extreme points into discriminative regions and extract deep features;

[0014] Step 6. Obtain fine-grained image recognition results.

[0015] Furthermore, the depth features are obtained by the following steps:

[0016] 1) Perform random horizontal flipping, scaling, and random cropping on the input image to meet the input requirements of the deep convolutional neural network;

[0017] 2) Input the transformed image into the convolutional neural network ResNet-50 to obtain the output features of its fourth stage, with a size of c*n*n, where c ≥ 1 is the number of channels of the feature and n ≥ 1 is the size of the feature map;

[0018] Furthermore, the self-attention map is obtained by the following steps:

[0019] 1) First, the feature map is scaled from c*n*n to c*n 2 ;

[0020] 2) Transpose the converted self-attention map and then perform matrix multiplication with itself to obtain a length of n 2 vector of

[0021] 3) Transform the vector to obtain a self-attention map of size n*n.

[0022] Furthermore, the input feature is recorded as X, the self-attention map is recorded as A, and the scale transformation operation is recorded as reshape. The process can be expressed as follows: A = reshape (X T X)

[0023] Furthermore, the class activation map is obtained by the following steps:

[0024] 1) Perform global mean pooling on the feature map to obtain a feature vector, which is then input into the classifier to obtain the prediction result;

[0025] 2) Extract the classification weight corresponding to the predicted category in the classifier, perform weighted summation on each channel of the feature, and obtain the category activation map.

[0026] Furthermore, the predicted category is recorded as i, and the classifier weight corresponding to the i-th category is recorded as w i , the class activation map is recorded as C, and the process of obtaining the class activation map is expressed as: C = ∑ i w i f i

[0027] Furthermore, candidate extreme points are obtained through the following steps:

[0028] 1) Perform weighted summation of the self-attention map and the class activation map to obtain saliency maps of different regions;

[0029] 2) Slide the saliency map to obtain the local maximum values ​​in different windows as candidate extreme points.

[0030] Furthermore, the process can be expressed by a formula S=αA+(1-α)C, and its size is n*n.

[0031] Furthermore, the candidate extreme points are expanded into discriminative regions and feature extraction is performed through the following steps:

[0032] 1) Use the set threshold to screen the candidate extreme points and select the extreme points that are greater than the candidate threshold;

[0033] 2) Smoothing the candidate points with a learnable Gaussian kernel to obtain several superimposed discriminative regions;

[0034] 3) The original features are weighted according to the discriminative regions and used as the discriminative features fed into the classifier.

[0035] Furthermore, the coordinates of the candidate extreme points are marked as (x, y), S x,y is the value of the candidate extreme point (x, y) on the attention map, δ is the set threshold, then only S is selected x,y The extreme points that are greater than the threshold δ are regarded as candidate extreme points.

[0036] Furthermore, the process of smoothing the i-th candidate extreme point using the learnable Gaussian kernel is expressed as:

[0037] Where β is the parameter of the Gaussian kernel, (x, y) is the coordinate of the candidate extreme point, S x,y is the value of the candidate extreme point (x, y) on the attention map.

[0038] Furthermore, the entire convolutional neural network is trained end-to-end using the cross-entropy loss function.

[0039] Furthermore, the fine-grained image recognition results are obtained through the following steps:

[0040] 1) Input the discriminative features into the multi-layer perceptron to obtain the probability distribution of each category;

[0041] 2) Use the softmax function to normalize each probability distribution;

[0042] 3) The corresponding category of the maximum normalized result is used as the fine-grained image recognition result.

[0043] Furthermore, by recognizing the accuracy As the evaluation index, n is the number of samples that are correctly identified and N is the total number of test samples.

[0044] Compared with the existing technology, the advantages of the present invention are:

[0045] 1. A saliency map acquisition method is proposed to generate candidate discriminative regions.

[0046] 2. Adaptively determine the number of discriminative regions to avoid the deterioration of fine-grained image recognition caused by missed or false detection of discriminative regions. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A diagram of the adaptive number of discriminative regions positioning and representation process of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described in detail below through specific embodiments and drawings.

[0049] The adaptive number of discriminative region positioning and representation methods of the present invention, such as Figure 1 As shown in Figure 2, it is mainly divided into two stages: training stage and testing stage.

[0050] The steps of the training phase are as follows:

[0051] Step 1: Train the image through a convolutional neural network to obtain extracted deep features for subsequent discriminative region positioning and representation.

[0052] The processing process of step 1 is as follows: first, the training image is passed through a convolutional neural network to obtain multi-level features of the training image. Since deep features are more suitable for representing high-level semantic information in the image, high-level features are extracted as features to be used. In the convolutional neural network ResNet-50 used in the present invention, for a fine-grained image with an input size of 448*448, the output feature size of ResNet-50 in the first stage is 128*128, which is reduced to 56*56 in the second stage, 28*28 in the third stage, and 14*14 in the last stage. The features of the last stage are used, and the size is 14*14.

[0053] Step 2. Based on the deep features obtained in step 1, a self-attention map is obtained using the self-attention mechanism to show the correlation between different regions, thereby helping to determine the discriminative regions.

[0054] Step 2 first transforms the obtained deep features into a self-attention mechanism to obtain a self-attention map of the same size. This process first transforms the deep features into a two-dimensional matrix c*n*n from a three-dimensional tensor c*n*n. 2 , c≥1, is the number of channels of the feature, n≥1, is the size of the feature map; then transpose it and multiply it with itself to get a matrix of length n 2The vector is then scaled back to its original size to obtain an n*n self-attention map. The self-attention map shows the relationship between each position on the feature. Positions with larger responses are often areas that are more important for classification.

[0055] Step 3. Calculate the class activation map for the predicted category based on the deep features to show the contribution of different regions to each category. This process extracts the prediction weights corresponding to the predicted category from the final linear classifier and performs a weighted summation of each channel of the original features to obtain the class activation map for the predicted category.

[0056] Step 4. Perform a weighted sum of the class activation map of the predicted class and the self-attention map of the feature to obtain a saliency map for locating the discriminative region.

[0057] Step 5. Slide the window on the saliency map obtained in step 4 to obtain candidate extreme points. After screening, the candidate extreme points are expanded into discriminative regions. Features are extracted from these discriminative regions to obtain discriminative features for classification.

[0058] First, the saliency map is windowed, and the maximum extreme point in each window is obtained as a candidate extreme point. The candidate extreme points are then screened, removing those with saliency values ​​below a threshold. Gaussian smoothing is then applied to the remaining candidate extreme points to obtain the discriminative regions corresponding to each candidate extreme point. Finally, the discriminative regions are superimposed and fused. This process sets a threshold, so the number of discriminative regions obtained is adaptive.

[0059] The discriminative region map S corresponding to the candidate extreme point (x, y) i,x,y for:

[0060]

[0061] Where β is the parameter of the Gaussian kernel, (x, y) is the coordinate of the candidate extreme point, S x,y is the value of the candidate extreme point (x, y) on the attention map.

[0062] Step 6. Obtain fine-grained image recognition results.

[0063] In step 6, the ResNet50 convolutional neural network performs global mean pooling on the discriminative regions to obtain a feature vector. This feature vector is then concatenated with the image feature vector obtained from the original image to obtain the feature vector used for classification. A multi-layer perceptron is used to obtain the probability distribution of each category, which is then normalized using the softmax function. The category with the largest output value is ultimately used as the image recognition result. The convolutional neural network uses an end-to-end training method, and the objective function uses the cross-entropy loss function.

[0064] The steps of the testing phase are as follows:

[0065] 1) Input the test image into the trained network, and the output category is the network recognition result.

[0066] 2) The test results are judged using the accuracy rate acc.

[0067] The test environment and experimental results of the adaptive number of discriminative region positioning and representation method proposed in this invention are as follows:

[0068] (1) Test environment:

[0069] System environment: Ubuntu 16.04;

[0070] Hardware environment: Memory: 16GB, GPU: TITIANXP, Hard disk: 1TB;

[0071] (2) Experimental data:

[0072] Training data:

[0073] The CUB-200-2011 dataset is used, which includes 5994 training images of 200 different bird categories. The training is carried out until the objective function converges and the model performance reaches a stable value.

[0074] Data preprocessing: Both training and test images were resized to 448 x 448 pixels using bilinear interpolation. During training, random cropping and random flipping were used for data augmentation.

[0075] Training optimization method: ADAM, with an initial learning rate of 0.01, which is reduced to 0.96 times the original learning rate after 4 epochs of training.

[0076] Test data: CUB-200-2011 test set (5794 images)

[0077] Evaluation method: Calculate the recognition accuracy of the test set

[0078] (3) Experimental results:

[0079] To illustrate the effect of the present invention, the fine classification recognition accuracy obtained by the method is verified on the CUB-200-2011 test set.

[0080] The results are shown in Table 1 below:

[0081] Table 1. Test results of the present invention

[0082] Serial number method Acc 1 The present invention 0.884

[0083] Acc refers to the accuracy rate.

[0084] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Those skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be based on the claims.

Claims

1. A method for locating and representing discriminative regions with adaptive quantity, characterized in that The steps include: Step 1. Extract deep features of the input image; Step 2. Use the self-attention mechanism on the deep features to obtain the self-attention map; Step 3. Extract category activation map for deep features; Step 4. Fuse the self-attention map and the class activation map to obtain a saliency map for locating discriminative regions. Step 5. Slide the window on the saliency map to obtain candidate extreme points, expand the candidate extreme points into discriminative regions, and extract discriminative features; Step 6. Obtain fine-grained image recognition results; The step 5 is specifically as follows: 1) Use the set threshold to filter the candidate extreme points and select the extreme points that are greater than the candidate threshold; 2) Smoothing the candidate points with a learnable Gaussian kernel to obtain several superimposed discriminative regions; 3) Weighting the original features according to the discriminative regions as the discriminative features fed into the classifier; The coordinates of the candidate extreme points are marked as (x, y), S x,y is the value of the candidate extreme point (x, y) on the attention map, δ is the set threshold, then only S is selected x,y The extreme points greater than the threshold δ are regarded as candidate extreme points; The process of smoothing the i-th candidate extreme point using a learnable Gaussian kernel is expressed as: Where β1 is the parameter of the Gaussian kernel.

2. The method for locating and representing discriminative regions with adaptive quantity according to claim 1, wherein: The step 1 is specifically as follows: 1) Perform random horizontal flipping, scaling, and random cropping on the input image; 2) Input the transformed image into the convolutional neural network ResNet-50 to obtain the output feature map with a size of c*n*n, where c ≥ 1 is the number of feature channels and n ≥ 1 is the size of the feature map.

3. The adaptive number discriminative region location and representation method according to claim 2, wherein: The step 2 is specifically as follows: 1) First, the feature map is scaled from c*n*n to c*n 2 ; 2) Transpose the converted self-attention map and then perform matrix multiplication with itself to obtain a length of n 2 vector of 3) Transform the vector to obtain a self-attention map of size n*n.

4. The method for locating and representing discriminative regions with adaptive quantity according to claim 3, wherein: The step 3 is specifically as follows: 1) Perform global mean pooling on the feature map to obtain a feature vector, which is then input into the classifier to obtain the prediction result; 2) Extract the classification weight corresponding to the predicted category in the classifier, perform weighted summation on each channel of the feature, and obtain the category activation map.

5. The method for locating and representing discriminative regions with adaptive quantity according to claim 4, wherein: The step 4 is specifically as follows: performing weighted summation on the self-attention map and the category activation map to obtain saliency maps of different regions.

6. The adaptive number discriminative region location and representation method according to claim 1, wherein: The step 6 is specifically as follows: 1) Input the discriminative features into the multi-layer perceptron to obtain the probability distribution of each category; 2) Use the softmax function to normalize each probability distribution; 3) The corresponding category of the maximum normalized result is used as the fine-grained image recognition result.

Images