A fine-grained image classification method based on data enhancement

Abstract

The application provides a fine-grained image recognition method based on data enhancement, and belongs to the technical field of computer vision. In the training stage, the most discriminative fine components of an object are located as enhanced component images through a feature map, and then the most discriminative fine components of the object are erased as enhanced complementary images. By inputting the component images, the model can learn the most discriminative fine components of the object, and by inputting the complementary images, the network can be forced to discover other discriminative fine components of the object. The method proposed in the present application first locates the most discriminative components of the object as the enhanced component images and then erases them, which improves the proportion of the most discriminative components in the image, and the erasing is performed under the guidance of the component mask, so that excessive noise is avoided during erasing and the accuracy of the erasing is ensured; the model can learn multiple discriminative components of the object, and the fine-grained image classification effect is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a fine-grained image classification method based on data enhancement. BACKGROUND

[0002] Fine-grained image classification is to distinguish different sub-classes in the same parent class, such as identifying different birds, dogs, cars and airplanes. Unlike conventional classification, fine-grained image classification task has the following two difficulties: (1) the visual difference of different classes often depends on the subtle parts of the object, for example, different sub-classes of birds may only be the difference of beak, crest and leg. (2) Lack of training data, since the labeling of fine-grained image data requires expert knowledge, which makes the size of fine-grained image dataset small, leading to overfitting of the model during training. The mainstream algorithm of fine-grained image classification can be divided into two categories: the first category is to directly improve the representation ability of image features, and the second category is the method based on locating objects or object parts.

[0003] Recently, the data enhancement method based on erasing has attracted more and more attention, and this kind of method is to erase part of the object area in the training process, so as to force the network to find other areas of the object. Cutout is the most representative work among them, which randomly erases a square area in the image. But this kind of method has the following problems: (1) the grid size or density of erasing is a hyperparameter, which is easy to erase the whole foreground in the case of small object scale and introduce too much noise; (2) erasing is completely random, without the guidance of high-level semantic information, and it cannot be guaranteed that the erased area belongs to the foreground. SUMMARY

[0004] The technical problem solved by the present application is to provide a fine-grained image classification method based on data enhancement, which can solve the problems of difficulty in determining the grid size during feature erasing and completely random erasing area in the prior art.

[0005] To solve the above technical problems, one technical solution adopted by the present application is to provide a fine-grained image classification method based on data enhancement.

[0006] The fine-grained image classification method based on data enhancement comprises the following steps:

[0007] S1 data preprocessing: obtaining training images by image preprocessing on original images;

[0008] S2 model training:

[0009] S21 inputting the training images into the backbone network to obtain features Figure One , and calculating the original loss of classification after the features Figure One passing through the full connection layer;

[0010] S22 according to the feature Figure One The feature positioning is performed on the training image to obtain a positioning object graph;

[0011] S23 inputting the positioning object graph into a backbone network to obtain a feature Figure Two , the feature Figure Two After classification through a full connection layer, a classification positioning loss is calculated;

[0012] S24 according to the feature Figure Two Data enhancement processing is performed on the positioning object graph to obtain a component image and a complementary image;

[0013] S25 sequentially inputting the component image and the complementary image into the backbone network and the full connection layer, and calculating a classification component loss and a complementary loss;

[0014] S26 taking a sum of the original loss, the positioning loss, the component loss and the complementary loss as a whole loss of the model, and updating model parameters through back propagation;

[0015] S27 repeating step S2 until the model converges or a specified number of iterations is reached;

[0016] S3 image classification:

[0017] S31 inputting an image to be classified after image preprocessing into a backbone network to obtain a feature Figure Three ;

[0018] S32 according to the feature Figure Three Feature positioning is performed on the image to be classified to obtain a positioning object image;

[0019] S33 inputting the positioning object image into the backbone network and the full connection layer to obtain a label category corresponding to the image to be classified;

[0020] The backbone network is a pre-trained model with image classification as a training task;

[0021] The backbone network and the full connection layer in steps S21, S23 and S25 share parameters.

[0022] Further, the image preprocessing includes image scaling, random horizontal flipping, color jittering and normalization processing;

[0023] The image scaling is to scale the original image to a uniform size.

[0024] Further, the feature positioning is performed on the training image according to the feature Figure One to obtain a positioning object graph, including:

[0025] summing the features in the channel direction Figure One to obtain an activation Figure One A;

[0026] computing a mean value of the activation Figure One as a first threshold λ;

[0027] computing an object mask M Figure One according to the first threshold and the activation obj , and the calculation formula is as follows:

[0028] ;

[0029] performing bilinear interpolation scaling on the object mask to the size of the training image, and then performing element-wise multiplication on the training image to obtain a located object map;

[0030] summing the features in the channel direction Figure One to obtain an activation ;

[0031] wherein A (x,y) is a value of the activation Figure One at a (x, y) position;

[0032] F∈R C x H x W represents a feature map output by a last convolutional layer of the backbone network, F i represents an image of the i-th channel;

[0033] C is a channel number of the feature map;

[0034] W and H are respectively a pixel width and a pixel height of the activation Figure One .

[0035] Further, the data augmentation processing on the located object map to obtain a part image and a complementary image according to the features Figure Two , comprises:

[0036] summing the features in the channel direction Figure Two to obtain an activation Figure Two ;

[0037] taking a value of the activation Figure Two in the HxWxk largest value as a second threshold θ;

[0038] computing a part mask M Figure Two according to the second threshold and the activation part , and the calculation formula is as follows:

[0039] ;

[0040] Bilinear interpolation of the part mask to the size of the training image and minimum bounding rectangle of non-zero values to get the bounding box;

[0041] According to the bounding box, the most discriminative part image is cropped from the positioning object image;

[0042] According to the part mask, a complementary mask is calculated, and the calculation formula is as follows:

[0043] ;

[0044] The complementary mask is bilinearly interpolated to the size of the training image, and then multiplied element by element with the training image to obtain a complementary image;

[0045] According to the bounding box, the most discriminative part image is cropped from the positioning object image, further comprising bilinearly interpolating the part image to the size of the positioning object image;

[0046] Wherein, k is a number in the interval (0, 1);

[0047] H, W are the pixel width and pixel height of the activation Figure Two respectively;

[0048] B(x, y) is the value of the activation Figure Two at the position (x, y);

[0049] M part(x,y) is the value of the part mask at the position (x, y);

[0050] M com(x,y) is the value of the complementary mask at the position (x, y).

[0051] Further, according to the feature Figure Three , the feature positioning of the image to be classified is performed to obtain a positioning object image, comprising:

[0052] According to the feature Figure Three , summing in the channel direction to obtain the activation Figure Three C;

[0053] The mean value of the activation Figure Three is calculated as a third threshold setting β;

[0054] According to the third threshold and the activation Figure Three , an object mask is calculated, and the calculation formula is as follows:

[0055] ;

[0056] The object mask is bilinearly interpolated and scaled to the size of the image to be classified, and is multiplied element by element with the image to be classified to obtain a located object map;

[0057] wherein C (x,y) is the value of the activation Figure Three at the (x, y) position.

[0058] M new(x,y) is the value of the object mask at the (x, y) position.

[0059] Further, the original loss of the classification, the positioning loss of the classification and the component loss of the classification and the complementary loss are calculated, and the loss functions used are all cross-entropy loss functions.

[0060] The beneficial effects of the present application are:

[0061] (1) The present application first locates the most discriminative components of the object as an enhanced component image through the feature map output by the last layer in the backbone network, and then erases the most discriminative components to obtain a complementary image. Through the component image and the complementary image, the model can learn multiple discriminative components of the object, fully utilizes the information in the data to learn the subtle discriminative components of the object, and improves the fine-grained image classification effect.

[0062] (2) The present application first locates the most discriminative components of the object as an enhanced component image and then erases them, improves the proportion of the most discriminative components in the image, and then erases them under the guidance of the component mask, avoids introducing too much noise during erasing, and ensures the accuracy of erasing. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 is a model structure diagram of a fine-grained image classification method based on data enhancement.

[0064] Figure 2 is a flowchart for obtaining a located object map according to a fine-grained image classification method based on data enhancement.

[0065] Figure 3 is a flowchart for data enhancement processing according to a fine-grained image classification method based on data enhancement. DETAILED DESCRIPTION

[0066] The preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings, so that the advantages and features of the present application can be more easily understood by those skilled in the art, and the scope of protection of the present application can be more clearly defined.

[0067] Please refer to Figure 1 , the embodiments of the present application include:

[0068] A fine-grained image classification method based on data enhancement, comprising:

[0069] S1 data preprocessing: image preprocessing is performed on the original image to obtain a training image;

[0070] S2 model training:

[0071] S21 inputting the training image into a backbone network to obtain a feature Figure One , the feature Figure One After classification by a fully connected layer, the original loss of classification is calculated;

[0072] S22 performing feature positioning on the training image according to the feature Figure One to obtain a positioning object map;

[0073] S23 inputting the positioning object map into a backbone network to obtain a feature Figure Two , the feature Figure Two After classification by a fully connected layer, the positioning loss of classification is calculated;

[0074] S24 performing data enhancement processing on the positioning object map according to the feature Figure Two to obtain a component image and a complementary image;

[0075] S25 sequentially inputting the component image and the complementary image into a backbone network and a fully connected layer, and calculating a component loss and a complementary loss of classification;

[0076] S26 taking the sum of the original loss, the positioning loss, the component loss and the complementary loss as the overall loss of the model, and updating the model parameters through back propagation;

[0077] S27 repeating step S2 until the model converges or a specified number of iterations is reached;

[0078] S3 image classification:

[0079] S31 inputting an image to be classified, which has been subjected to image preprocessing, into a backbone network to obtain a feature Figure Three ;

[0080] S32 performing feature positioning on the image to be classified according to the feature Figure Three to obtain a positioning object image;

[0081] S33 inputting the positioning object image into a backbone network and a fully connected layer to obtain a label category corresponding to the image to be classified;

[0082] The backbone network is a pre-trained model with image classification as a training task;

[0083] The backbone network, the fully connected layer in the steps S21, S23 and S25 are shared parameters.

[0084] In the embodiment, ResNet-50 pre-trained on ImageNet classification dataset is used as the backbone network, and the feature map F of the last convolutional layer of ResNet-50 is taken as the input of the model for classification, the channel number C is 1024, the pixel height H is 14, and the pixel width W is 14. CxHxW The bird fine-grained dataset CUB-200-2011 is used for experiment.

[0085] As shown in Figure 1 , the model in the application has four branches: an original branch, a positioning branch, a component branch and a complementary branch, which are used to calculate the original loss, the positioning loss, the component loss and the complementary loss of classification respectively. In the training stage of the model, the four branches are trained at the same time, and in the inference stage of the model, only the positioning branch is used, without the need for data enhancement processing and without the need to calculate the original loss, the component loss and the complementary loss of classification.

[0086] Further, the image preprocessing includes image scaling, random horizontal flipping, color jittering and normalization processing.

[0087] The image scaling is to scale the original image to a unified size.

[0088] In the embodiment, before the images in the dataset are input into the model, image preprocessing is performed to enhance the dataset, expand the sample number of the dataset and improve the generalization ability of the model, including: image scaling, which scales the image to a unified size, in the embodiment, the image is scaled to 448x448; random horizontal flipping, which increases the diversity of the image; color jittering, which generates a new image by randomly adjusting the saturation, brightness and contrast of the image; normalization processing, which includes maximum and minimum normalization, dividing each value in the image by 255, dividing each value in the image by 127.5 and then subtracting 1, etc., and in the embodiment, maximum and minimum normalization is used.

[0089] Further, as shown in Figure 2 , the feature Figure One is used to perform feature positioning on the training image to obtain a positioning object map.

[0090] The feature Figure One is used to perform feature positioning on the training image to obtain a positioning object map. Figure One A;

[0091] The mean value of the activation Figure One is calculated as a first threshold value λ.

[0092] The first threshold value and the activation Figure OneComputing the object mask M obj , the calculation formula is as follows:

[0093] ;

[0094] Bilinearly interpolating the object mask to the size of the training image, and then element-wise multiplying the training image to obtain a located object map;

[0095] The feature Figure One The calculation formula for summing in the channel direction is ;

[0096] Wherein, A (x,y) is the value of the activation Figure One at the (x, y) position;

[0097] F∈R C x H x W represents the feature map output by the last convolution layer of the backbone network, F i represents the i-th channel image;

[0098] Element-wise multiplication means that two images with the same dimensions, i.e. value matrices, are multiplied at corresponding positions to finally obtain a matrix with the same dimensions as the original image.

[0099] Further, the feature Figure Two According to the data enhancement processing of the located object map, a part image and a complementary image are obtained, including:

[0100] The feature Figure Two is summed in the channel direction to obtain the activation Figure Two ;

[0101] Taking the HxWxk largest value in the activation Figure Two as the second threshold θ;

[0102] According to the second threshold and the activation Figure Two , a part mask M part is calculated, and the calculation formula is as follows:

[0103] ;

[0104] Bilinearly interpolating the part mask to the size of the training image and then calculating the minimum bounding rectangle of the non-zero value to obtain a bounding box;

[0105] According to the bounding box, the most discriminative part image is cropped from the located object map;

[0106] According to the part mask, a complementary mask is calculated, and the calculation formula is as follows:

[0107] ;

[0108] performing bilinear interpolation scaling on the complementary mask to a size of the training image, and multiplying the complementary mask obtained by the training image element by element to obtain a complementary image;

[0109] The cutting out of the most discriminative part image from the positioning object image according to the bounding box further comprises bilinear interpolation of the part image to the size of the positioning object image.

[0110] Wherein, k is a number in the interval (0, 1);

[0111] H, W are respectively the pixel width and the pixel height of the activation Figure Two ;

[0112] B(x, y) is the value of the activation Figure Two at the position (x, y);

[0113] M part(x,y) is the value of the part mask at the position (x, y);

[0114] M com(x,y) is the value of the complementary mask at the position (x, y).

[0115] In this embodiment, for the bird fine-grained dataset CUB-200-2011, k = 0.25.

[0116] The value of k can be selected according to the proportion of the object in the image in the dataset. The larger the proportion of the object in the image, the larger the value of k.

[0117] Figure 3 is a schematic diagram of feature positioning of the training image according to the feature Figure One in the embodiment.

[0118] Further, the feature positioning of the image to be classified according to the feature Figure Three obtains a positioning object image, comprising:

[0119] Summing the feature Figure Three in the channel direction to obtain the activation Figure Three C;

[0120] Calculating the mean of the activation Figure Three as a third threshold setting β;

[0121] According to the third threshold and the activation Figure Three , the object mask is calculated, and the calculation formula is as follows:

[0122] ;

[0123] Bilinearly interpolating and scaling the object mask to the size of the image to be classified, and multiplying the image to be classified element by element to obtain a located object map;

[0124] wherein C (x,y) is the value of the activation Figure Three at the (x, y) position;

[0125] M new(x,y) is the value of the object mask at the (x, y) position.

[0126] Further, the original loss of the calculation classification, the calculation classification of the positioning loss and the calculation classification of the component loss and the complementary loss, the loss function used is all cross entropy loss function.

[0127] According to the characteristics Figure Two The data enhancement processing of the located object map obtains a component image and a complementary image, which can be replaced by the training image and the characteristics Figure One respectively instead of the located object map and the characteristics Figure Two , and the training loss of the model training is the sum of the original loss, the component loss and the complementary loss.

[0128] In the fine-grained image classification task, some target subjects have a small area ratio in the image, and some target subjects have a large area ratio in the image; when the image is subjected to data enhancement processing, the located object map and the characteristics Figure Two are subjected to data enhancement processing, which can solve the problem of small area ratio of the target subject in the image and more noise in the image, and for the case that the area ratio of the target subject in the image is large, the feature positioning step can be directly skipped, and the training image and the characteristics Figure One are subjected to data enhancement processing, that is, the training image and the characteristics Figure One are respectively replaced by the located object map and the characteristics Figure Two to perform fine-grained image classification.

[0129] The above only describes the embodiments of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A fine-grained image classification method based on data augmentation, characterized in that, The method comprises the following steps: S1 data preprocessing: performing image preprocessing on the original image to obtain a training image; S2 model training: S21 inputting the training image into a backbone network to obtain a feature map one, and calculating an original loss of classification after the feature map one is classified by a fully connected layer; S22 performing feature positioning on the training image according to the feature map one to obtain a positioning object map; S23 inputting the positioning object map into the backbone network to obtain a feature map two, and calculating a positioning loss of classification after the feature map two is classified by the fully connected layer; S24 performing data enhancement processing on the positioning object map according to the feature map two to obtain a component image and a complementary image; S25 sequentially inputting the component image and the complementary image into the backbone network and the fully connected layer, and calculating a component loss of classification and a complementary loss of classification; S26 taking the sum of the original loss, the positioning loss, the component loss and the complementary loss as the overall loss of the model, and updating the model parameters through back propagation; S27 repeating step S2 until the model converges or a specified number of iterations is reached; S3 image classification: S31 inputting an image to be classified, which has been subjected to image preprocessing, into a backbone network to obtain a feature map three; S32 performing feature positioning on the image to be classified according to the feature map three to obtain a positioning object image; S33 inputting the positioning object image into the backbone network and the fully connected layer to obtain a label category corresponding to the image to be classified; The backbone network is a pre-trained model with image classification as the training task; The backbone network and the fully connected layer in steps S21, S23 and S25 share parameters.

2. The method of claim 1, wherein the data augmentation is based on, The image preprocessing comprises image scaling, random horizontal flipping, color jittering and normalization processing; The image scaling is to scale the original image to a uniform size.

3. The data augmentation based fine-grained image classification method of claim 2, wherein, The feature positioning on the training image according to the feature map one comprises: summing the feature map one in the channel direction to obtain an activation map one A; calculating the mean value of the activation map one as a first threshold value λ; According to the first threshold and the activation map, a mask M of the object is calculated obj The calculation formula is as follows: ； performing bilinear interpolation scaling on the object mask to the size of the training image, and then multiplying the object mask with the training image element by element to obtain a positioning object map; The calculation formula of summing the feature map in the channel direction is ; where A (x,y) is the value of the activation map at (x, y) position; F ∈ R C x H x W denotes the feature map output by the last convolutional layer of the backbone network, F i denotes the image of the i-th channel; C is the number of channels of the feature map; W and H are the pixel width and the pixel height of the activation map one respectively.

4. The method of claim 3, wherein the data augmentation is based on a combination of the following: rotation, translation, scaling, and flipping. The data enhancement processing on the positioning object map according to the feature map two comprises: summing the feature map two in the channel direction to obtain an activation map two; taking the HxWxkth largest value in the activation map two as a second threshold value θ; According to the second threshold and the activation map two computing component mask M part , the calculation formula is as follows: ； performing bilinear interpolation on the component mask to the size of the training image, and then calculating the minimum bounding rectangle of the non-zero values to obtain a bounding box; cutting out a component image with the most discriminative ability from the positioning object map according to the bounding box; calculating a complementary mask according to the component mask, and the calculation formula is as follows: ； performing bilinear interpolation scaling on the complementary mask to the size of the training image, and then multiplying the complementary mask with the training image element by element to obtain a complementary image; The most discriminative part image is cropped from the positioning object image according to the bounding box, and the part image is bilinearly interpolated to the size of the positioning object image; Wherein, k is a number in the interval (0, 1); H, W are the pixel width and height of the activation map two respectively; B(x, y) is the value of the activation map two at the position (x, y); M part(x,y) is the value of the component mask at the (x, y) location; M com(x,y) is the value of the complementary mask at the (x, y) position.

5. The method of claim 4, wherein the data augmentation is based on, The feature positioning of the image to be classified according to the feature map three includes: Summing the feature map three in the channel direction to obtain an activation map three C; The mean value of the activation map three is calculated as a third threshold setting β; According to the third threshold and the activation map three, an object mask is calculated, and the calculation formula is as follows: ； The object mask is bilinearly interpolated and scaled to the size of the image to be classified, and is multiplied element by element with the image to be classified to obtain a positioning object image; where C (x,y) is the value at position (x,y) in the activation map three; M new(x,y) is the value of the object mask at the (x, y) position.

6. The method of claim 1, wherein the data augmentation is based on, The loss function used for calculating the original loss of classification, calculating the positioning loss of classification and calculating the part loss and complementary loss of classification is a cross entropy loss function.

7. A fine-grained image classification method employing the method of claim 1, characterized by, The training image and the feature map one are respectively replaced by the positioning object image and the feature map two.

Images