Network data supervised fine-grained image recognition method based on attention data augmentation

Abstract

The present invention discloses a network data supervised fine-grained image recognition method based on attention data enhancement, which comprises the following steps: 1) using the fine-grained category of an image as a keyword to retrieve images on the network, generating sample images according to the fine-grained category corresponding to the image, and using each sample image to perform warm-up training on a fine-grained image recognition model; 2) selecting a group of the sample images and inputting them into the fine-grained image recognition model after warm-up training respectively to obtain a category activation map corresponding to each sample image; 3) generating an attention map corresponding to the sample image according to the category activation map of the sample image; 4) randomly selecting two sample images, mixing the images and labels according to their attention maps, and obtaining an enhanced sample image; 5) using the sample images after data enhancement to train a fine-grained image recognition model; 6) using the trained fine-grained image recognition model to recognize an image to be recognized, and obtaining the fine-grained category in the image to be recognized.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to a network data enhancement method and a fine-grained image recognition method based on an attention mechanism, which can learn to recognize fine-grained images from data with noisy labels crawled from the network. Background Art

[0002] Fine-grained image recognition is a critical research task in computer vision, with a wide range of applications and research needs in both industry and academia. Fine-grained image recognition tasks require identifying specific subcategories within a broad category, such as bird species or car models. The difficulty of fine-grained image recognition lies in its narrow inter-class differences and large intra-class variance. On the one hand, compared to traditional image recognition tasks, fine-grained categories are very similar, with only minor differences. On the other hand, images within the same category can exhibit significant differences due to factors such as shooting angle, lighting, and growth cycle.

[0003] In recent years, deep learning methods for fine-grained image recognition have emerged in large numbers. However, the success of these deep learning methods often requires large amounts of labeled training data. Due to the subtle differences between fine-grained categories, labeling training data often requires domain experts, which is costly. To reduce the cost of manual labeling, researchers have proposed many methods, primarily focusing on semi-supervised learning and small-sample learning. However, these methods inevitably require a portion of labeled data as prior knowledge for training. To further reduce labeling costs, directly using images of relevant categories crawled from the web for training has attracted widespread attention.

[0004] Unlike manually annotated datasets, images scraped from the web often contain a large amount of noisy labels due to a lack of careful scrutiny and processing. This means that the image's true category is inconsistent with the assigned label. Research has shown that deep neural networks have strong fitting capabilities. Training directly with scraped data can cause the model to memorize data with incorrect labels, thereby reducing its generalization ability.

[0005] Traditional methods for processing noisy label data can be divided into the following four categories:

[0006] 1. Design new loss functions that are robust to noise, such as mean absolute error

[0007] 2. Estimate the noise transfer matrix and post-process the model prediction results through the transfer probability

[0008] 3. Design a regularization method that is robust to noise, such as the mixup series

[0009] 4. Filter out clean samples with correct labels for training. In the early stages of training, samples with small loss are often clean samples.

[0010] Most existing methods are a combination of the above methods. Mixup, a data augmentation method for mixed samples, is widely used in supervised learning of network data due to its contribution to model generalization performance. However, most existing sample mixing methods only interpolate the labels of the mixed regions based on the area of the image, ignoring the importance of the mixed regions to the entire image. For example, if the cat region in one image completely covers the dog region in another, the mixed image should be completely classified as a cat, rather than a weighted sum of the two labels based on the covered area. Therefore, it is necessary to study a method for sample mixing based on the importance of the mixed regions. Summary of the Invention

[0011] Aiming at fine-grained network supervised images in the real world, this paper proposes a network data augmentation method based on the attention mechanism to generate training data and train the model, thereby improving the model's robustness to noisy labels and achieving the purpose of learning to recognize fine-grained images directly from network images.

[0012] The present invention is a general data augmentation method that uses a class activation map to construct an attention map. It then blends any two images according to their attention maps, blends the labels according to the importance of the blended regions, and finally uses the blended images and labels to train the model.

[0013] The technical solution adopted in the present invention is as follows:

[0014] A network data enhancement method and fine-grained image recognition method based on an attention mechanism, comprising the following steps:

[0015] 1) Perform 5-10 epochs of warm-up training on the fine-grained image recognition model. The fine-grained image recognition model consists of a convolutional neural network for feature extraction and a linear classification layer for classification. During warm-up training, the fine-grained categories are used as keywords to retrieve relevant images from the web to generate training data. The fine-grained image recognition model is then trained using a standard cross-entropy loss function. If the convolutional neural network used (such as ResNet-50) has been pre-trained on ImageNet, the corresponding pre-trained parameters can be used directly, fixing these parameters and pre-training only the linear classification layer. The number of warm-up training epochs can be adjusted based on the specific situation. To prevent the model from overfitting to noisy labels in the network image data, the number of epochs should be limited.

[0016] 2) Use the convolutional neural network in the fine-grained image recognition model to extract features from a set of network sample images.

[0017] 3) For each input sample image, the feature map (Featuremap) output by the last layer of the convolutional neural network and the weights of the linear classification layer are used to generate the category activation map corresponding to the sample image, and the attention map is generated based on the category activation map.

[0018] 4) Randomly select two sample images and mix the images and labels according to their attention maps to obtain a sample image and its label, completing data enhancement.

[0019] 5) Use the mixed images and labels to train a fine-grained image recognition model.

[0020] 6) Using the trained fine-grained image recognition model to identify the image to be identified, and obtaining the fine-grained categories in the image.

[0021] Furthermore, the entire fine-grained image recognition model consists of a convolutional neural network for feature extraction and a linear classification layer for classification.

[0022] Furthermore, convolutional neural networks can be models of arbitrary structures.

[0023] Furthermore, the class activation map is generated by the feature map output by the last layer of the convolutional neural network and the weights of its linear classification layer, and then enlarged to the same size as the input sample image through bilinear interpolation. The specific principle of the class activation map is as follows:

[0024]

[0025] where f∈R C×H×W is the feature map output by the last layer of the convolutional neural network, w∈R C is the classification weight vector corresponding to the sample image label category in the linear classification layer, H and W are the height and width of the feature map, c represents the cth channel of the feature map, C is the total number of channels, and f c ∈R H×W is the feature data corresponding to the cth channel in the feature map f, w c ∈R is the scalar at the cth position in the classification weight vector w. Specifically, the weight of the linear classification layer is where N c is the total number of categories in the training set, w∈R used to generate the category activation map Cis the C-dimensional vector corresponding to the sample image label category. By weighted summing the feature maps according to the classification weights corresponding to the sample image label category, a class activation map for the label category can be constructed to show the importance of different regions in the image to the label category when classifying. However, the size of the generated class activation map is often smaller than the sample image size, so it needs to be bilinearly interpolated to enlarge it to the sample image size. At the same time, in order to perform the binarization operation below, the interpolated class activation map needs to be maximum and minimum normalized. The specific operation is as follows:

[0026]

[0027] Where i and j represent the horizontal and vertical coordinates of the pixel point, CAM ij Represents the pixel value of the pixel corresponding to the i-th row and j-th column in the class activation map CAM; min(CAM) and max(CAM) represent the minimum and maximum values in the class activation map CAM, respectively.

[0028] Furthermore, after obtaining the category activation map of each sample, a threshold σ is set to binarize it and generate an attention map. The specific operations are as follows:

[0029]

[0030] Where M is the attention map, i and j represent the horizontal and vertical coordinates of the pixel points, and M ij Represents the pixel value of the pixel corresponding to the i-th row and j-th column in the attention map M; the area with a pixel value of 1 in the attention map M represents the activated area.

[0031] Furthermore, the sample images are mixed using the attention map obtained by binarization. The specific operation is to perform a weighted sum of the activation area in one image and the same area in another image according to the importance of the area to the two images. First, the image I is calculated. a The activation area in image I a Importance:

[0032]

[0033] Among them, M a For image I a Attention map, CAM a For image I a Class activation map of ; Represents the attention map M a The pixel value of the pixel corresponding to the i-th row and j-th column in , Represents the attention map CAM a The pixel value of the pixel point corresponding to the i-th row and j-th column in the image. b The same region in image Ib Importance:

[0034]

[0035] Then, according to the importance, a and I b To mix:

[0036]

[0037] Among them I mix The image after mixing is mixed with I a Mainly, I a The activation area in I b The same area in the weighted summation is performed while maintaining I a The inactive area remains unchanged. a The activation area corresponds to I b The method will reject the inactive area of I b Mixing is done to eliminate I b Background information on I a The impact of foreground information; specifically, when I a The activation area corresponds to I b In the inactive area, w cor =0, so According to I mix The calculation formula shows that the mixed image is still I a .

[0038] Furthermore, the image labels are mixed using the binary attention map. The specific operation is to perform a weighted sum of the labels of the two images according to the importance of the activated area and the inactivated area to the two images. The weight is divided into three parts. The first part is the image I a Importance of non-activated regions:

[0039] λ1=1-w a

[0040] The second part is the image I in the activation area after image mixing. a Proportion:

[0041]

[0042] The second part is the image I in the activation area after image mixing. b Proportion:

[0043]

[0044] Finally, the labels of the two images are weighted summed according to the three weights:

[0045] y mix =(λ1+λ2)*y a +λ3*y b

[0046] where y mix is the mixed label, y a and y b Represents image I a and I b The original label is a one-hot vector.

[0047] Further, using the mixed image I mix With its label y mix The model is trained in an end-to-end manner. The loss function is the cross entropy loss function. The overall objective function is:

[0048]

[0049] Where N is the number of training samples, i is the i-th training sample, is the mixed label of the i-th training sample, y i It is the prediction result of the fine-grained image recognition model for the i-th training sample.

[0050] The evaluation index of the network data supervised fine-grained image recognition method is the recognition accuracy, which is calculated as follows:

[0051]

[0052] Where n is the number of samples identified correctly, and N is the total number of test samples.

[0053] The trained model is then used to identify the image to be recognized, obtaining fine-grained categories within the image. This is the testing and practical application phase of the model. During this phase, the image to be recognized is fed into the model, where it passes through a convolutional neural network to generate a feature map. This feature map is then subjected to a global average pooling operation to generate a feature vector. The linear classification layer takes the feature vector as input and outputs a predicted probability distribution, selecting the category with the highest predicted probability as the image's category.

[0054] In summary, the present invention designs a network data-supervised fine-grained image recognition method based on attention data enhancement, which improves the classification effect of the model. Compared with existing technologies, the advantages of the present invention are:

[0055] 1. An attention-based data augmentation method is proposed to reduce the impact of noisy label data on the model and improve the generalization ability of the model.

[0056] 2. In data enhancement, the importance of the image mixed area to the two images is taken into account at the same time. This method will reduce the impact of the background information of one image on the foreground information of another image, which is more reasonable.

[0057] 3. In network-supervised fine-grained image recognition, the classification accuracy of the model is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Framework diagram of the fine-grained image recognition model supervised by network data based on attention data enhancement. DETAILED DESCRIPTION

[0059] The present invention is further described in detail below through specific embodiments and drawings.

[0060] The process of the network data supervised fine-grained image recognition method based on deep learning of the present invention is as follows: Figure 1 As shown in Figure 2, it is mainly divided into two stages: training stage and testing stage.

[0061] The steps of the training phase are as follows:

[0062] 1) Use the parameters of a ResNet-50 pre-trained on ImageNet as the initial model parameters. Modify the final linear classification layer output to 200 classes. Fix the convolutional neural network portion and perform a five-epoch warm-up training on the linear classification layer using the training images. Because the linear classification layer parameters are required to generate the class activation map, directly using the initialized parameters will degrade the class activation map quality. Due to the characteristics of deep neural networks, the model initially learns clean labeled samples and gradually fits to noisy labeled data over time. Therefore, a small number of epochs of warm-up training will not learn too much noisy labeled data.

[0063] 2) The training image passes through the convolutional neural network to obtain the category activation map, and then performs interpolation and normalization operations for the subsequent data enhancement process:

[0064] The processing process of step 2) is as follows: the training image is input into the convolutional neural network ResNet-50 used in the present invention. For an image with an input size of 224*224, the feature map size output by the last convolutional layer of ResNet-50 is 7*7, and the calculated category activation map size is also 7*7. In order to supervise the entire input image, the 7*7 category activation map is enlarged to 224*224 by bilinear interpolation. In order to generate an attention map according to the threshold later, the pixel value of the category activation map is normalized to [0,1] using maximum and minimum normalization.

[0065] 3) Use the category activation map to generate an attention map, perform data augmentation on the input image according to the attention map, and obtain the enhanced image and label.

[0066] In step 3), a threshold σ is set to binarize the class activation map to generate an attention map. Pixel values in the class activation map that are greater than the threshold are set to 1, and otherwise are set to 0. The region with a pixel value of 1 is the activation region, which plays an important role in identifying the image label category during the classification process. The activation region of one image is mixed with the same region of another image to reduce the model's attention to background information. The importance of the activation region to both images needs to be considered during mixing:

[0067]

[0068]

[0069] Perform a weighted sum of the activated regions according to their importance, while keeping the inactivated regions unchanged:

[0070]

[0071] The mixed image needs to be relabeled. This label is obtained by weighted summing of the labels of the two mixed images. Specifically, it is the sum of the labels of three parts. The first is the label of the inactive area:

[0072] y1=(1-w a )*y a

[0073] Next are the labels contributed by the two images in the activation region:

[0074]

[0075]

[0076] The final label of the mixed image is the sum of y1, y2 and y3:

[0077] y mix =y1+y2+y3

[0078] 4) The enhanced images and labels (I mix ,y mix ) is re-input into the convolutional neural network for training.

[0079] The convolutional neural network used in step 4) is the same as the convolutional neural network in step 2). The data-augmented image is input into the convolutional neural network for feature extraction to obtain a feature map. Then, a feature vector is obtained through global average pooling. Finally, a linear layer is input for classification to obtain the probability distribution of each category, and then the Softmax function is used to normalize it. Figure 1 The GAP in the above sentence refers to the global average pooling operation, which converts the extracted feature maps into feature vectors, making them easier to input into a linear classifier for classification. The model uses an end-to-end training method, and the objective function uses the cross-entropy loss function.

[0080] The steps of the testing phase are as follows:

[0081] 1) The test image is directly input into the trained model without data augmentation, and the category with the highest output probability is the model recognition result.

[0082] 2) The test results are judged using the accuracy rate acc.

[0083] The data enhancement method of the present invention is further described below. The attention map in the data enhancement method proposed in the present invention can be generated by any spatial attention mechanism, and a threshold needs to be set to binarize the attention map to extract the activation area for image blending.

[0084] The test environment and experimental results of the network data-supervised fine-grained image recognition method proposed in this paper based on attention data enhancement are as follows:

[0085] (1) Test environment:

[0086] System environment: Windows 10;

[0087] Hardware environment: Memory: 64GB, GPU: RTX3090, Hard disk: 2TB;

[0088] (2) Experimental data:

[0089] Training data:

[0090] Using the Web-bird dataset, the training set includes 18,388 pictures with noisy labels crawled from the Internet, totaling 200 bird species.

[0091] Data preprocessing: Both training and test images were resized to 224x224 using bilinear interpolation. During training, random cropping and random flipping were used for data augmentation.

[0092] Training optimization method: SGD, with an initial learning rate of 0.01, training for 200 epochs, reducing the learning rate to 0.1 times the original learning rate after 50 epochs, setting the threshold σ to 0.8, and using ResNet-50 pre-trained on ImageNet as the initial model.

[0093] Test data: Web-bird test set (5794 images)

[0094] Evaluation method: Calculate the recognition accuracy of the test set

[0095] (3) Experimental results:

[0096] In order to illustrate the effect of the present invention, the same data set is used to test whether the attention data enhancement method proposed in the present invention is used or not, and the Web-bird test set is used.

[0097] The test comparison results of the existing invention are shown in Table 1 below:

[0098] Table 1. Comparison of test results between existing methods and the present invention

[0099] Serial number method Acc 1 The present invention (without using attention data enhancement) 0.7188 2 The present invention (using attention data enhancement) 0.7361

[0100] As can be clearly seen from the table, the network data supervised fine-grained image recognition method based on attention data enhancement involved in the present invention shows a significant improvement compared to the results without attention data enhancement, which demonstrates the effectiveness of the present invention.

[0101] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Those skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be based on the claims.

Claims

1. A network data-supervised fine-grained image recognition method based on attention data augmentation, comprising the following steps: 1) Using the fine-grained categories of images as keywords to retrieve relevant images on the Internet, generating a sample image based on each retrieved image and its corresponding fine-grained category, and using each of the sample images to perform warm-up training on the fine-grained image recognition model; 2) Selecting a group of sample images and inputting them into the fine-grained image recognition model after warm-up training in step 1) to obtain a category activation map corresponding to each sample image; 3) Generate the attention map of the corresponding sample image based on the category activation map of the sample image; 4) Randomly select two sample images, and mix the images and labels of the two selected sample images according to their attention maps to obtain a data-enhanced sample image; 5) using the data-enhanced sample images to train the fine-grained image recognition model; 6) Using the fine-grained image recognition model trained in step 5), the image to be recognized is recognized to obtain the fine-grained category in the image to be recognized.

2. The method according to claim 1, characterized in that The fine-grained image recognition model includes a convolutional neural network for feature extraction and a linear classification layer for classification; for each input sample image, the feature map output by the last layer of the convolutional neural network and the weights of the linear classification layer are used to generate a category activation map corresponding to the input sample image.

3. The method according to claim 2, characterized in that The class activation map is enlarged to the same size as the input sample image by bilinear interpolation.

4. The method according to claim 2 or 3, characterized in that Using the formula CAM = ∑ 1≤c≤C w c f c Generate the category activation map; where f c is the feature data corresponding to the cth channel in the feature map, w c is the scalar at the cth position in the classification weight vector w, C is the total number of channels, and CAM is the class activation map.

5. The method according to claim 1, 2 or 3, characterized in that: The class activation map is binarized by setting a threshold σ to generate the attention map.

6. The method according to claim 5, characterized in that The pixel value of the pixel corresponding to the i-th row and j-th column in the attention map Among them, CAM ij The pixel value of the pixel corresponding to the i-th row and j-th column in the category activation image.

7. The method according to claim 1, 2 or 3, characterized in that: Randomly select two sample images a and b, I a is the image in the sample image a, M a For image I a Attention map, CAM a For image I a The class activation map of Represents the attention map M a The pixel value of the pixel corresponding to the i-th row and j-th column in , Represents the attention map CAM a The pixel value of the pixel point corresponding to the i-th row and j-th column in I b is the image in the sample image b, M b For image I b Attention map, CAM b For image I b The class activation map of Represents the attention map M b The pixel value of the pixel corresponding to the i-th row and j-th column in , Represents the attention map CAM b The pixel value of the pixel corresponding to the i-th row and j-th column in ; The method for obtaining the data-enhanced sample image based on the selected sample images a and b is: 41) Calculate image I a The activation area in image I a Importance Calculate Image I b The same region in image I b Importance 42) According to the importance of I a and I b mix Get the mixed image I mix ; 43) Use the binary attention map to mix the image labels y mix =(λ1+λ2)*y a +λ3*y b , get the mixed label y mix ; Among them, y a and y b Represents image I a and I b The original label, λ1=1-w a , 44) By image I mix and its corresponding label y mix The sample images after data enhancement corresponding to the selected sample images a and b.

8. The method according to claim 7, characterized in that The loss function used when training the fine-grained image recognition model using the sample images after data augmentation is Among them, N is the total number of sample images after data enhancement, is the label of the sample image after data augmentation of the i-th data, y i It is the prediction result of the fine-grained image recognition model for the sample image after data enhancement.

9. A server, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images