A fine-grained image classification method and system based on feature correction

Abstract

The application relates to a fine-grained image classification method and system based on feature correction. The image classification method comprises the following steps: S1, preprocessing each image in a training set to obtain a pretreated image; S2, constructing a network model based on feature correction, extracting features F L and feature F L‑1 from the pretreated image; S3, removing common features between classes to increase inter-class variance to obtain a first classification loss; S4, reducing intra-class variance by using a center loss to obtain a second classification loss; S5, obtaining an overall loss by weighted summation, and training the network model by using a back propagation method; and S6, predicting the class of each image to be processed by using the trained network model. By extracting common prototypes of the image to be processed and removing common similar features, the inter-class variance is increased, the intra-class variance is reduced by using a center loss, and the network is guided to obtain better classification results, so that the accuracy of fine-grained image classification is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to a fine-grained image classification method, in particular to a fine-grained image classification method based on feature correction and a fine-grained image classification system based on feature correction. BACKGROUND

[0002] In recent years, with the development of deep learning, convolutional neural network has become one of the most important feature extraction methods. At present, convolutional neural network has been widely used in image classification, natural language processing, target detection and other fields.

[0003] Traditional image classification tends to classify different kinds of objects (such as cats, dogs, etc.), but the classification results of the same kind of species (such as Black_footed_Albatross (black-footed albatross), Canada_Warbler (Canada warbler), etc.) are relatively poor, so fine-grained visual classification emerges as the times require and becomes a research hotspot. Fine-grained visual classification is a very challenging research topic in the field of computer vision. Simply put, its purpose is to reclassify some basic large categories (for example, birds, cars, airplanes, etc.). Compared with traditional visual classification, due to the influence of factors such as posture, light, occlusion, background, the inter-class difference of fine-grained images is smaller, but the intra-class difference is larger, which makes fine-grained visual classification more challenging.

[0004] In fine-grained image classification, the difference between classes may only be in very local positions such as eyes or beaks, so it is crucial to find the most discriminative local features. In fine-grained image classification, there are usually two ways to find local discriminative features: (1) supervised learning method; (2) weakly supervised learning method. The supervised learning method usually uses manual labeling of local discriminative blocks to assist the network in learning. However, the use of manual labeling wastes a lot of manpower and material resources, and is difficult to apply in real-world scenarios. Therefore, people turn their attention to weakly supervised learning methods. The weakly supervised learning method does not require manual labeling of local discriminative blocks, but uses the learning ability of the network to automatically locate the position of the local discriminative feature, saving a lot of manpower and material resources, and being more suitable for real-world scenarios. The weakly supervised learning method can usually be divided into two ways: localization classification and attention. The localization classification method usually locates the position of the most discriminative feature first, and then uses methods such as cropping and enlargement to enlarge the local feature and input it into the network for further learning. However, the localization classification method is more complex to train, and may be in multiple stages. The attention method was first inspired by the way people look at things. People can quickly find the thing they want from a pile of things because the human brain consciously or unconsciously selects important information and filters out other useless information. Inspired by this idea, some predecessors use the network to automatically locate and learn the most discriminative part. However, due to the large intra-class variance and small inter-class variance of fine-grained images, the attention mechanism may not be able to well locate the local discriminative feature. SUMMARY

[0005] Therefore, it is necessary to provide a fine-grained image classification method and system based on feature correction in view of the problem that the large intra-class variance and small inter-class variance in the existing fine-grained image classification result in low classification accuracy.

[0006] The application is implemented by the following technical solutions: a fine-grained image classification method based on feature correction comprises the following steps:

[0007] S1: pre-processing each image in the training set to obtain a corresponding pre-processed image.

[0008] S2: constructing a network model based on feature correction, extracting the Lth stage feature F L and the (L-1)th stage feature F L-1 of the pre-processed image through the backbone network of the network model.

[0009] S3: increasing the inter-class variance by removing the common similar features between classes to obtain a first classification loss. The method for increasing the inter-class variance is as follows:

[0010] S31: input the feature F L-1 The input channel attention model generates a channel attention map.

[0011] S32: input the channel attention map into a spatial attention model to generate a spatial attention map.

[0012] S33: extract all common prototypes P of all classes in all preprocessed images, and learn common similar features F between classes in the common prototypes comm .

[0013] S34: remove the common similar features from the spatial attention map to obtain inter-class corrected features F.

[0014] S35: input the inter-class corrected features F into a convolutional block to obtain discriminative features F5.

[0015] S36: input the discriminative features F5 into a first classifier to obtain a first classification loss L cls1 . The first classification loss L cls1 is expressed as:

[0016] L cls1 = -∑y·log[C1(GMP(F5))

[0017] In the formula, C1(.) represents the first classifier for classification, GMP(.) represents global maximum pooling, and y represents the true value label of the image.

[0018] S4: adopt a center loss to reduce intra-class variance to obtain a second classification loss. The method for reducing intra-class variance is as follows:

[0019] S41: input the feature F L into an attention convolutional block to obtain an attention feature F attention .

[0020] S42: perform bilinear fusion of the attention feature F attention ={f1, f2, f3, …, f M} and the feature F L to obtain a bilinear feature B. The bilinear feature B is expressed as:

[0021]

[0022] In the formula, g(.) represents global maximum pooling or global average pooling, <.> represents a concate operation, is a bilinear feature, M is the number of channels of the attention feature, and C L is the number of channels of the Lth stage.

[0023] S43: the difference between the bilinear feature B and the class center feature is minimized by using the center loss, and then the bilinear feature is input into the second classifier to obtain a second classification loss L cls2 . Wherein, the center loss L center is expressed as:

[0024]

[0025] In the formula, N represents the total number of label categories, k={1, 2, 3, …, N}, represents the L2 norm, represents the class center feature vector of the kth class, B k is the bilinear feature of the kth class.

[0026] The second classification loss L cls2 is expressed as:

[0027] L cls2 =-∑y·log[C2(GMP(F L ))]

[0028] In the formula, C2(.) represents the second classifier used for classification.

[0029] S5: the first classification loss, the second classification loss and the center loss are weighted and summed to obtain the overall loss L total , and the network model is trained by using the method of back propagation. Wherein, the overall loss L total is expressed as:

[0030] L total =αL cls1 +βL cls2 +L center

[0031] In the formula, α and β are both artificial set hyperparameters.

[0032] S6: all the images to be classified in the fine-grained data set are input into the trained network model, and the predicted class corresponding to each image to be processed is obtained, and then the fine-grained data set is classified according to the predicted class. The class y * of the image to be classified is expressed as:

[0033]

[0034] In the formula, argmax(.) is a function of finding the maximum value of parameters.

[0035] The image classification method extracts common prototypes of the training set images, learns common similar features between classes in the common prototypes, then increases the inter-class variance by removing the common similar features, and reduces the intra-class variance by using a center loss to guide the network to obtain better classification results and improve the accuracy of fine-grained image classification.

[0036] In one of the embodiments, the generation method of the channel attention map is as follows:

[0037] S311: The feature F is input into a global average pooling layer and a global maximum pooling layer to obtain two channel description operators. L-1 Global average pooling and global maximum pooling are respectively performed to obtain two channel description operators.

[0038] S312: The two channel description operators are input into a multi-layer perceptron to obtain two enhanced channel description operators.

[0039] S313: The two enhanced channel description operators are fused by element-wise addition, and an activation function is used to obtain a channel weight A c .

[0040] S314: The channel weight A c is multiplied element-wise with the feature F L-1 to obtain a channel attention map. The channel attention map F c is expressed as:

[0041]

[0042] In the formula, represents element-wise multiplication.

[0043] In one of the embodiments, the channel weight A c is expressed as:

[0044] A c =σ(MLP(avgpool(F L-1 ))+MLP(maxpool(F L-1 )))

[0045] In the formula, sigma(.) is a sigmoid(.) activation function, MLP(.) is a perceptron, avgpool(.) is average pooling, and maxpool(.) is maximum pooling.

[0046] In one of the embodiments, the generation method of the spatial attention map is as follows:

[0047] S321: The channel attention map is subjected to channel-based global average pooling and global maximum pooling to obtain two spatial description operators.

[0048] S322: input the two spatial description operators after splicing and convolution into an activation function to obtain a spatial weight A s .

[0049] S323: element-wise multiply the spatial weight and the channel attention map to obtain a spatial attention map F s . The spatial attention map F s is expressed as:

[0050]

[0051] In one of the embodiments, the spatial weight A s is expressed as:

[0052] A s = σ(conv([avgpool(F c ); max pool(F c )]))

[0053] In the formula, conv(.) represents convolution.

[0054] In one of the embodiments, the picture information of the preprocessed image is subjected to an average operation to obtain a common prototype P. The common prototype P is expressed as:

[0055]

[0056] In the formula, c represents the number of categories of the fine-grained dataset, m represents the number of images in each category, represents the jth image of the ith category, i = {1, 2, 3, …, c}, and j = {1, 2, 3, …, m}.

[0057] In one of the embodiments, a convolution operation is used to extract common similar features from the common prototype P. The common similar features F comm are expressed as:

[0058] F comm = f(P)

[0059] In the formula, f(.) represents a convolution network.

[0060] In one of the embodiments, the spatial attention map is subjected to a subtraction operation with the common features, and then input into a Relu function to obtain inter-class correction features. The inter-class correction features F are expressed as:

[0061]

[0062] In the formula, relu(.) represents a Relu activation function, epoch is the number of training times, and n is a hyperparameter of the number of training times.

[0063] The application further provides a fine-grained image classification system based on feature correction, which comprises a common prototype extraction module and an image classification network model.

[0064] The common prototype extraction module is used for pre-processing images to be classified in a training set to obtain pre-processed images, and then performing average operation according to picture information of the pre-processed images to obtain common prototypes.

[0065] The image classification network model comprises a backbone network, a feature correction module and a prediction output module. L and feature F L-1 The feature correction module comprises an inter-class variance increasing module and an intra-class variance reducing module. L The inter-class variance increasing module is used for removing common similar features between classes to increase inter-class variance and obtain a corresponding first classification loss. The intra-class variance reducing module is used for bilinear fusion of feature F

[0066] In one of the embodiments, the inter-class variance increasing module comprises a channel attention submodule, a spatial attention submodule, a discriminative feature extraction submodule and a first classifier. L-1 The channel attention submodule is used for inputting the pooled feature F L-1 into a multilayer perceptron to obtain corresponding channel weights, and then performing element-wise multiplication of the channel weights and feature F The spatial attention submodule is used for pooling, concatenating and convolving the channel attention map, and then passing the channel attention map through an activation function to obtain corresponding spatial weights, and performing element-wise multiplication of the spatial weights and the channel attention map to obtain a spatial attention map. The discriminative feature extraction submodule is used for convolving the common prototypes to obtain common similar features, and then removing the common similar features from the spatial attention map to obtain inter-class correction features, and convolving the inter-class correction features to obtain discriminative features. The first classifier is used for training the discriminative features by using cross-entropy loss to obtain the first classification loss.

[0067] Compared with the prior art, the application has the following beneficial effects:

[0068] 1. The application extracts common prototypes of a training set, simultaneously learns common similar features between classes in the common prototypes, then increases inter-class variance by removing the common similar features, and reduces intra-class variance by using center loss, so as to guide the network to obtain better classification results and improve the accuracy of fine-grained image classification. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1A step diagram of the feature correction-based fine-grained image classification method of Embodiment 1 of the present application;

[0070] Figure 2 A process diagram of fine-grained classification of bird images using the feature correction-based network model;

[0071] Figure 3 A structure diagram of the feature correction-based fine-grained image classification system of Embodiment 1 of the present application;

[0072] Figure 4 A structure diagram of the feature correction-based fine-grained image classification system of Embodiment 1 of the present application; Figure 3 A structure diagram of the medium channel attention sub-module;

[0073] Figure 5 A structure diagram of the medium channel attention sub-module; Figure 3 A structure diagram of the medium spatial attention sub-module. DETAILED DESCRIPTION

[0074] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0075] It should be noted that when a component is referred to as being "mounted on" another component, it can be directly on the other component or there can be a middle component. When a component is referred to as being "disposed on" another component, it can be directly disposed on the other component or there can be a middle component. When a component is referred to as being "fixed on" another component, it can be directly fixed on the other component or there can be a middle component.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terminology used in the description of the present application herein only for the purpose of describing specific embodiments and is not intended to limit the present application. The term "or / and" used herein includes any and all combinations of one or more related listed items.

[0077] Embodiment 1

[0078] Please refer to Figure 1 , which is a step diagram of the feature correction-based fine-grained image classification method of Embodiment 1 of the present application. The feature correction-based fine-grained image classification method includes:

[0079] S1: Preprocess each image in the training set to obtain a corresponding preprocessed image. The preprocessing method includes random cropping and random horizontal flipping. In this embodiment, the user first divides the fine-grained image set to be classified into a training set and a test set. Then, all images in the training set are preprocessed by random cropping and random horizontal flipping. The size of the preprocessed image after random cropping is 448x448.

[0080] S2: Construct a network model based on feature correction, and extract the Lth stage feature F L and the (L-1) th stage feature F L-1 The network model includes a backbone network, a correction network, and a prediction network. The correction network includes a class variance increasing module and a class variance decreasing module.

[0081] In this embodiment, the backbone network uses Resnet-50, C L-1 , H L-1 , and WL L-1 are the channel number, height, and width of the L-1 stage feature, C L , H L , and W L are the channel number, height, and width of the L stage feature.

[0082] ResNet50 is a commonly used deep convolutional neural network. Its structure includes 5 stages:

[0083] Stage 1: contains 1 convolutional layer and 1 pooling layer, mainly responsible for feature extraction of the input preprocessed image.

[0084] Stage 2: contains 3 convolutional layers and 2 pooling layers, mainly responsible for further deepening and improving the features extracted in the first stage.

[0085] Stage 3: contains 4 convolutional layers and 2 pooling layers, mainly responsible for further deepening and improving the features extracted in the second stage.

[0086] Stage 4: contains 6 convolutional layers and 2 pooling layers, mainly responsible for further deepening and improving the features extracted in the third stage.

[0087] Stage 5: contains 3 convolutional layers and 1 pooling layer, mainly responsible for further deepening and improving the features extracted in the fourth stage.

[0088] In summary, the main role of these 5 stages is to extract and deepen the features of the input image for subsequent image classification.

[0089] S3: Increase the inter-class variance by removing the common similar features between classes to obtain a first classification loss. In fine-grained image classification, due to the large intra-class gap and small inter-class gap, the classification is more difficult. Therefore, it is important to find the most discriminative local features. Since the features between classes are too similar, the positioning of the local features is not very accurate, which further affects the accuracy of classification. Therefore, an inter-class variance increasing module is designed to learn common features, and the inter-class discriminativeness is increased by removing common features.

[0090] The method for increasing the inter-class variance is as follows:

[0091] S31: The feature F L-1 is input into a channel attention model to learn the relationship between channels and generate a channel attention map. The specific method is as follows:

[0092] S311: The feature F L-1 is respectively subjected to global average pooling and global maximum pooling to obtain two 1x1xC L-1 channel description operators. Wherein, C L-1 is the channel number of the feature of the L-1 stage.

[0093] S312: The two channel description operators are input into a multi-layer perceptron to obtain two enhanced channel description operators.

[0094] S313: The two enhanced channel description operators are fused by element-wise addition, and the channel weight A c is obtained by sigmoid activation function. The channel weight A c is expressed as:

[0095] A c =σ(MLP(avgpool(F L-1 ))+MLP(maxpool(F L-1 )))

[0096] Wherein, σ(.) is a sigmoid activation function, MLP(.) is a multi-layer perceptron, avgpool(.) is an average pooling, and maxpool(.) is a maximum pooling.

[0097] S314: The channel weight A c is multiplied with the feature F L-1 element by element to obtain a channel attention map. The channel attention map F c is expressed as:

[0098]

[0099] Wherein, represents element-wise multiplication.

[0100] S32: input the channel attention map as an input feature map into the spatial attention model to learn the relationship between pixels in the feature space, and then obtain a spatial attention map. The specific method is as follows:

[0101] S321: perform channel-based global average pooling and global maximum pooling on the channel attention map to obtain two H L-1 ×W L-1 ×1 spatial description operators.

[0102] S322: perform channel-based concatenation operation on the two spatial description operators, then perform a convolution operation on the concatenated spatial description operator, and then pass it through a sigmoid activation function to obtain a spatial weight A s . The spatial weight A s is expressed as:

[0103] A s =σ(conv([avgpool(F c );maxpool(F c )]))

[0104] In the formula, conv(.) represents convolution.

[0105] S323: multiply the spatial weight and the channel attention map element by element to obtain a spatial attention map F s . The spatial attention map F s is expressed as:

[0106]

[0107] Although the attention model can further locate the local features, due to the great similarity between classes in fine-grained image classification, the positioning of local features causes some errors, which will cause certain difficulties in fine-grained image classification. Therefore, common similar features can be extracted from the public prototype, which makes the classification of images more difficult. By removing the common similar features in the discriminative features, the similarity between classes is reduced. S33: perform average operation on the picture information of the preprocessed image to obtain the public prototype of all classes. Learn the common similar features between classes in the public prototype. Specifically, perform average operation on the extracted picture information of all preprocessed images to obtain the public prototype of all classes. And save all classes of the public prototype as a “.pth” format file. Of course, in other embodiments, the size of the preprocessed image can be larger or smaller, and the public prototype can also use other formats such as “.pt”, “.pkl” and the like. The public prototype P can be expressed as:

[0108]

[0109] where c denotes the number of classes of fine-grained dataset, m denotes the number of images in each class, denotes the jth image of the ith class, i = {1, 2, 3, …, c}, j = {1, 2, 3, …, m}.

[0110] The convolution operation is performed on the common prototype P of all classes to extract the common similar features between classes. Then the common similar features F comm are expressed as:

[0111] F comm = f(P)

[0112] where f(.) denotes a convolutional network.

[0113] S34: The common similar features F are removed from the spatial attention map to obtain the inter-class correction features F, so as to reduce the similarity between classes. Specifically, the spatial attention map F s is subtracted from the common similar features F comm , and at the same time, the original features F L-1 are added in the first n epochs. Finally, the inter-class correction features F are obtained after the Relu activation function. The inter-class correction features F can be expressed as:

[0114]

[0115] where relu(.) denotes the Relu activation function, epoch is the number of training times, and n is the hyperparameter of the number of training times.

[0116] S35: The inter-class correction features F are input into a convolutional block for further learning to obtain the accurate discriminative features F5. The discriminative features F5 are expressed as:

[0117] F5 = conv_block(F)

[0118] where conv_block(.) is a convolutional block.

[0119] S36: The discriminative features F5 are input into the first classifier to obtain the first classification loss L cls1 . The first classifier includes global maximum pooling and multi-layer perceptron. First, the most discriminative features on each channel are obtained by using global maximum pooling, then the channel number is changed to the class number by using the full connection layer to represent the probability of belonging to each class, and finally the cross-entropy loss is used for training to obtain the first classification loss. The first classification loss L cls1 is expressed as:

[0120] Lcls1 = -∑y·log[C1(GMP(F5))]

[0121] In the formula, C1(.) represents the first classifier used for classification, GMP(.) represents global max pooling, and y represents the ground truth label of the image.

[0122] Please combine Figure 2 This diagram illustrates the process of using a feature-based network model for fine-grained classification of bird images. Specifically, Figure 2 In this process, preprocessed bird images are first input into a network model. The backbone network extracts the corresponding basic bird features (the feature regions learned by the backbone network are marked with bounding boxes). Simultaneously, common similarity features between bird species are extracted from common class prototypes. Then, a correction network removes these common similarity features from the basic bird features, yielding the corresponding discriminative features. These discriminative features, as more refined classification features, can guide the image to be classified into the correct category. For example... Figure 2 As shown, a feature-based network model is used to correctly classify the input bird image as a yellow-breasted giant bird. Yellow-breasted Chat.

[0123] S4: Employ central loss on feature F L To reduce the intra-class variance, a second classification loss is obtained. The feature F... L The data is fed into the within-class variance reduction module. This module first utilizes bilinear pooling to increase the feature F. L The dimension enriches the feature F L The representation allows for more thorough learning of detailed features. Then, center loss is used to reduce intra-class variance, further improving model accuracy. Finally, bilinear features are fed into the second classifier to obtain the second classification loss. The specific methods for reducing intra-class variance are as follows:

[0124] S41: Feature F L An attention feature F is obtained by inputting an attention convolutional block. attention Then the attention feature F attention Expressed as,

[0125] F attention =conv_att(F L )

[0126] In the formula, conv_att(.) represents a convolution operation with a kernel size of 1×1. M is the number of channels for the attention feature.

[0127] S42: Incorporate attention features F attention With feature FL The bilinear fusion is performed to obtain bilinear features B.

[0128] F attention = {f1, f2, f3, …, f M} and F L The bilinear pooling is performed to perform fusion and learn some detailed features. The specific formula is as follows:

[0129]

[0130] In the formula, g(.) represents global maximum pooling or global average pooling, <.> represents a concate (fusion) operation, is a bilinear feature, C L is the number of channels of the Lth stage.

[0131] S43: The center loss minimizes the difference between the bilinear feature B and the class center feature vector, and then the bilinear feature is input into the second classifier to obtain the second classification loss L cls2 . By reducing the difference between the bilinear feature B and the class center feature vector, the difference between the features in the class is reduced. Wherein, the center loss L center is expressed as:

[0132]

[0133] In the formula, N represents the total number of label categories, k = {1, 2, 3, …, N}, represents an L2 norm, represents the class center feature vector of the kth class, B k is the bilinear feature of the kth class.

[0134] C k is initialized by a zero vector, and is updated by a moving average method during the training process, and the formula is as follows:

[0135] C k ← (1-β) C k + βB k

[0136] In the formula, β is an artificial set hyperparameter.

[0137] All images in each training iteration process help to update the center feature vector. The second classifier also includes global maximum pooling and a multilayer perceptron. First, the most discriminative features on each channel are obtained by using global maximum pooling, and then the channel number is changed to the number of categories by a fully connected layer, so as to represent the probability of belonging to each category. Finally, the cross-entropy loss is used for training to obtain the second classification loss. The second classification loss L cls2 is expressed as:

[0138] L cls2 = -∑y·log[C2(GMP(F L ))

[0139] In the formula, C2(.) represents the second classifier for classification.

[0140] S5: The first classification loss, the second classification loss and the center loss are weighted and summed to obtain the overall loss L total , and the network model is trained by using the method of back propagation. The overall loss L total is expressed as:

[0141] L total = αL cls1 + βL cls2 + L center

[0142] In the formula, α is an artificial set hyperparameter.

[0143] Back propagation is to calculate the gradient of the loss function of all weights in the network model, and the gradient is fed back to the optimization method to update the weight value to minimize the loss function. The back propagation algorithm first calculates the output value of each node in the forward propagation manner, and then calculates the partial derivative of the loss function value with respect to each parameter in the reverse propagation traversal graph manner.

[0144] In the embodiment, after obtaining the final overall loss, SGD (stochastic gradient descent) is used as the optimizer of the network, the learning rate is set to 0.001, and the batchsize (sample number) is set to 8. Then, the network model is trained by using back propagation.

[0145] S6: All images to be classified in the fine-grained data set are input into the trained network model to obtain the predicted category corresponding to each image to be classified, and then the fine-grained data set is classified according to the predicted category. After the network model is trained, the trained model is saved. Then, the pictures in the test set are sent into the saved model. During the test, only the center loss in the intra-class variance reduction module is removed. If the test result meets the classification accuracy, the corresponding network model can be saved.

[0146] The results of using two classifiers are used for prediction, and the category y * of the image to be processed for prediction is expressed as:

[0147]

[0148] In the formula, argmax(.) is a function of finding the maximum value of the parameter.

[0149] The feature correction-based fine-grained image classification method of the embodiment first extracts a common prototype of an image to be processed, and simultaneously learns common similar features between classes in the common prototype, then increases the variance between classes by removing the common similar features, and reduces the variance within classes by using a center loss, so as to guide the network to obtain better classification results and improve the accuracy of fine-grained image classification.

[0150] Experimental verification

[0151] The image classification method of the embodiment is used for further classification of bird image sets, vehicle image sets and aircraft image sets, and other existing classification methods are used as references. Specifically, for each image set, a training set and a test set are divided, ResNet-50 is used to extract features of images in all training sets, each classification network model is trained, and the final training parameters are saved. Then, images in the test set are input into each classification network model to obtain the classification results of each image set. The final classification results are shown in Table 1.

[0152] Table 1: Comparison of classification results

[0153]

[0154] As can be seen from Table 1, when fine-grained classification is performed on birds and vehicles, the final classification accuracy of the image classification method of the embodiment is obviously higher than that of other classification methods. When fine-grained classification is performed on aircraft images, the classification accuracy of the image classification method of the embodiment is only slightly lower than that of DP-Net, is consistent with that of DF-GMM, and is obviously higher than that of other classification methods. In summary, for the same image set, when the same backbone network is used for feature extraction, the image classification method of the embodiment obviously has higher classification accuracy.

[0155] Referring to Figure 3 , which is a structural schematic diagram of the feature correction-based fine-grained image classification system of the embodiment. In order to implement the above-mentioned image classification method and facilitate accurate classification of fine-grained images, the application further provides a feature correction-based fine-grained image classification system, which comprises a common prototype extraction module and an image classification network model.

[0156] The common prototype extraction module is used for pre-processing the images to be classified in the training set to obtain pre-processed images, and then performing average operation according to the picture information of the pre-processed images to obtain the common prototype.

[0157] The image classification network model comprises a backbone network, a feature correction module and a prediction output module. The backbone network is used for extracting features F L and F L-1 of the common prototype. In the embodiment, the backbone network adopts Resnet-50, C L-1 , H L-1 and W L-1 are respectively the number of channels, height and width of the L-1 stage feature, C L , H L and W L are respectively the number of channels, height and width of the L stage feature.

[0158] The feature correction module comprises an inter-class variance increasing module and an intra-class variance reducing module. The inter-class feature increasing module is used to remove common similar features between classes to increase the inter-class variance, and obtain a corresponding first classification loss. The inter-class variance increasing module comprises a channel attention submodule, a spatial attention submodule, a discriminative feature extraction submodule and a first classifier.

[0159] Please refer to Figure 4 , which is a structural diagram of the channel attention submodule in Figure 3 . The channel attention submodule is used to input a multi-layer perceptron after pooling the feature F L-1 to obtain the corresponding channel weight, and then multiply the channel weight with the feature F L-1 element by element to form a channel attention map. Specifically, first, the feature F L-1 is respectively globally averaged and globally maximally pooled to obtain two 1x1xC L-1 channel description operators, and the two channel description operators are input into a multi-layer perceptron to obtain two enhanced channel description operators. Then, the two enhanced channel description operators are fused by element-wise addition, and the channel weight A c is obtained after sigmoid activation. Finally, the channel weight A c is multiplied with the feature F L-1 element by element to obtain the channel attention map.

[0160] Please refer to Figure 5 , which is a structural diagram of the spatial attention submodule in Figure 3 . The spatial attention submodule is used to pool, concatenate and convolve the channel attention map, and obtain the corresponding spatial weight through an activation function, and then multiply the spatial weight with the channel attention map element by element to obtain a spatial attention map. Specifically, first, the channel attention map is respectively globally averaged and globally maximally pooled based on the channel to obtain two H L-1 xW L-1 x1 spatial description operators. Then, the two spatial description operators are concatenated based on the channel, and then the concatenated spatial description operators are convolved, and then the sigmoid activation function is used to obtain the spatial weight A sFinally, the spatial weight and the channel attention map are multiplied element by element to obtain the spatial attention map F s .

[0161] The discriminative feature extraction submodule is used to convolve the public prototype to obtain the public similar feature, and then remove the public similar feature from the spatial attention map to obtain the inter-class correction feature. Specifically, the spatial attention map is subtracted from the public similar feature, and at the same time, the original feature F L-1 is added in the first n epochs. After passing through the Relu activation function, the inter-class correction feature is obtained. Finally, the inter-class correction feature is convolved to obtain the discriminative feature. By removing the inter-class public similar feature, the similarity between classes can be reduced.

[0162] The first classifier is used to train the first classification loss according to the discriminative feature using the cross-entropy loss. Specifically, the first classifier includes global maximum pooling and a multi-layer perceptron. First, the most discriminative feature on each channel is obtained using global maximum pooling, and then the channel number is changed to the class number through the fully connected layer to represent the probability of belonging to each class. Finally, the first classification loss is obtained by training using the cross-entropy loss.

[0163] The intra-class variance reduction module is used to perform bilinear fusion on the feature F L , and minimize the difference between the bilinear feature and the class center feature vector to reduce the intra-class variance, to obtain the corresponding second classification loss. Specifically, the intra-class variance reduction module can include an attention convolution block, a bilinear fusion submodule, and a second classifier. First, the feature F L is input into the attention convolution block to obtain an attention feature. Then, the attention feature and the feature F L are input into the bilinear fusion submodule to obtain a bilinear feature. Finally, the center loss is used to remove the class center feature vector in the bilinear feature, and then the bilinear feature is input into the second classifier to obtain the second classification loss.

[0164] The prediction output module is used to save the parameters of the trained model that meets the test accuracy, and then output the predicted class according to the input image to be classified. Specifically, after the network model is trained, the parameters of the trained model are saved. Then, the pictures in the test set are input into the saved network model. During testing, the center loss in the intra-class variance reduction module is removed. If the test result meets the classification accuracy, the corresponding network model parameters can be saved. The image to be classified is input into the saved network model to obtain the predicted class.

[0165] Any combination of the technical features in the above-described embodiments can be made, and for the sake of brevity, not all possible combinations are described, however, as long as the combination of the technical features does not exist in contradiction, it shall be considered within the scope of the present disclosure.

[0166] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it shall not be understood as a limitation on the patent scope of the present application. It shall be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these shall be within the protection scope of the present application. Therefore, the protection scope of the present application patent shall be subject to the appended claims.

Claims

1. A fine-grained image classification method based on feature correction, which achieves accurate classification of images in fine-grained image classification by increasing inter-class variance and decreasing intra-class variance; characterized in that, The fine-grained image classification includes the following steps: S1: Preprocess each image in the training set to obtain the corresponding preprocessed image; S2: Construct a feature-corrected network model, and extract the L-th stage features F of the preprocessed image through the backbone network of the network model. L and the characteristics of stage L-1 F L-1 ; S3: Increase the inter-class variance by removing common similarity features between classes to obtain the first classification loss; the specific methods for increasing the inter-class variance are as follows: S31: Feature F L-1 The input channel attention model generates a channel attention map; S32: Input the channel attention map into the spatial attention model to generate a spatial attention map; S33: Extract the common prototype of all classes in all the preprocessed images; learn the common similarity feature F between classes from the common prototype. comm ; S34: Remove common similarity features from the spatial attention map to obtain inter-class modified features F; S35: Input the inter-class correction feature F into a convolutional block to obtain the discriminative feature F5; S36: Input the discriminative feature F5 into the first classifier to obtain the first classification loss L. cls1 The first classification loss L cls1 Expressed as: L cls1 =-∑y·log[C1(GMP(F5))] In the formula, C1(.) represents the first classifier used for classification, GMP(.) represents global max pooling, and y represents the ground truth label of the image; S4: Use central loss to reduce within-class variance to obtain the second classification loss; the specific method for reducing within-class variance is as follows: S41: Feature F L An attention feature F is obtained by inputting an attention convolutional block. attention ; S42: Transfer the attention feature F attention ={f1,f2,f3,…,f M } and the feature F L Bilinear fusion is performed to obtain bilinear feature B; the bilinear feature B is expressed as: In the formula, g(.) represents global max pooling or global average pooling, and <.> represents the concatenation operation. It is a bilinear feature, M is the number of channels for the attention feature, and C L It is the number of channels in stage L; S43: Minimize the difference between the bilinear feature B and the class center feature using center loss, and then input the bilinear feature into the second classifier to obtain the second classification loss L. cls2 Wherein, the central loss L center Expressed as: In the formula, N represents the total number of label categories, and k = {1, 2, 3, ..., N}. Represents the L2 norm. B represents the class center feature vector of the k-th class. k It is a bilinear feature of the k-th class; Second classification loss L cls2 Expressed as: L cls2 =-Σy·log[C2(GMP(F L ))] In the formula, C2(.) represents the second classifier used for classification; S5: The overall loss L is obtained by weighted summation of the first category loss, the second category loss, and the centrality loss. total The network model is trained using backpropagation; wherein the overall loss L total Expressed as: L total =αL cls1 +βL cls2 +L center In the formula, α and β are both artificially set hyperparameters; S6: Input the images to be classified from the fine-grained dataset into the trained network model to obtain the predicted category corresponding to each image to be processed, and then classify the fine-grained dataset according to the predicted category; the category y of the image to be classified * Represented as: In the formula, argmax(.) is a function that calculates the maximum value of the parameter.

2. The fine-grained image classification method based on feature correction according to claim 1, characterized in that, In step S31, the method for generating the channel attention map is as follows: S311: For feature F L-1 Two channel descriptor operators are obtained by performing global average pooling and global max pooling respectively; S312: Input the two channel description operators into a multilayer perceptron to obtain two corresponding enhanced channel description operators; S313: The two enhanced channel descriptors are fused element-wise, and the channel weights A are obtained after passing through an activation function. c ; S314: Set the channel weight A c With feature F L-1 The channel attention map is obtained by performing element-wise multiplication; the channel attention map F c Expressed as: In the formula, This indicates element-wise multiplication.

3. The fine-grained image classification method based on feature correction according to claim 2, characterized in that, In step S313, the channel weight A c Expressed as: A c <σ(MLP(avgpool(F L-1 ))+MLP(maxpool(F L-1 ))) In the formula, σ(.) is the sigmoid activation function, MLP(.) is the perceptron, avgpool(.) is the average pooling, and maxpool(.) is the max pooling.

4. The fine-grained image classification method based on feature correction according to claim 1, characterized in that, In step S32, the method for generating the spatial attention map is as follows: S321: Perform channel-based global average pooling and global max pooling on the channel attention map to obtain two spatial descriptor operators; S322: Concatenate and convolve the two spatial descriptor operators, then input them into the activation function to obtain the spatial weights A. s ; S323: Multiply the spatial weights and the channel attention map element-wise to obtain the spatial attention map F. s The spatial attention map F s Expressed as:

5. The fine-grained image classification method based on feature correction according to claim 4, characterized in that, In step S322, the spatial weight A s Expressed as: A s =σ(conv([avgpool(F c );maxpool(F c )])) In the formula, conv(.) represents convolution.

6. The fine-grained image classification method based on feature correction according to claim 1, characterized in that, In step S33, the image information of the preprocessed image is averaged to obtain a common prototype; the common prototype P is expressed as: In the formula, c represents the number of categories in the fine-grained dataset, and m represents the number of images in each category. Let i represent the j-th image of the i-th class, where i = {1, 2, 3, ..., c} and j = {1, 2, 3, ..., m}.

7. The fine-grained image classification method based on feature correction according to claim 1, characterized in that, In step S33, a convolution operation is used to extract the common similarity features from the common prototype P; the common similarity features F comm Expressed as: F comm =f(P) In the formula, f(.) represents a convolutional network.

8. The fine-grained image classification method based on feature correction according to claim 1, characterized in that, In step S34, the spatial attention map is subtracted from the common similarity feature, and then input into the ReLU function to obtain the inter-class correction feature; the inter-class correction feature F is expressed as: In the formula, relu(.) represents the ReLU activation function, epoch is the number of training iterations, and n is the hyperparameter for the number of training iterations.

9. A fine-grained image classification system based on feature correction, which employs the fine-grained image classification method based on feature correction as described in any one of claims 1 to 8, characterized in that, The image classification system includes: The common prototype extraction module is used to preprocess the images to be classified in the training set to obtain preprocessed images, and then perform an average calculation based on the image information of the preprocessed images to obtain the common prototype. An image classification network model includes a backbone network, a feature correction module, and a prediction output module; the backbone network is used to extract features F. L and feature F L-1 The feature correction module includes an inter-class variance increasing module and an intra-class variance decreasing module. The inter-class variance increasing module is used to remove common similar features between classes to increase the inter-class variance and obtain the corresponding first classification loss. The intra-class variance decreasing module is used to adjust the feature F... L Bilinear fusion is performed, and the difference between the fused features and the class center feature vector is minimized to reduce the intra-class variance, thus obtaining the corresponding second classification loss; the prediction output module is used to output the predicted category based on the input image to be classified.

10. The fine-grained image classification system based on feature correction according to claim 9, characterized in that, The module for increasing inter-class variance includes a channel attention submodule, a spatial attention submodule, a discriminative feature extraction submodule, and a first classifier; the channel attention submodule is used to process the feature F L-1 After pooling, the input is given to a multilayer perceptron to obtain the corresponding channel weights. These channel weights are then compared with the feature F. L-1 Perform element-wise multiplication and merge to form a channel attention map; The spatial attention submodule is used to pool, concatenate, and convolve the channel attention map, and obtain the corresponding spatial weights through an activation function. The spatial weights are then multiplied element-wise with the channel attention map to obtain the spatial attention map. The discriminative feature extraction submodule is used to convolve the common prototype to obtain common similarity features, and then remove the common similarity features from the spatial attention map to obtain inter-class correction features. The inter-class correction features are then convolved to obtain discriminative features. The first classifier is used to obtain a first classification loss by training with cross-entropy loss based on the discriminative features.

Images