A fine-grained image classification method based on convolutional neural network

Abstract

The application discloses a fine-grained image classification method based on a convolutional neural network, and belongs to the technical field of fine-grained image processing. First, a classification network model is constructed by adopting a fusion channel feature re-attention module and a spatial multi-region feature attention module, then a contrast learning loss term in a loss function is designed by adopting the idea of contrast learning, and finally the classification network model is used to classify images obtained in real time. Specifically, the method comprises the following steps: constructing a classification network model; the classification network model comprises a feature extraction network, a channel feature re-attention module, a spatial multi-region feature attention module and a classifier; constructing a training data set and performing model training; obtaining images to be classified in real time, feeding the images into the trained classification network model, and obtaining classification results of the current images. The application effectively reduces the difficulty of fine-grained image classification and solves the limitation of the attention mechanism in the field.

Description

Technical Field

[0001] This invention belongs to the field of fine-grained image processing technology, specifically relating to a fine-grained image classification method based on convolutional neural networks. Background Technology

[0002] In recent years, deep learning has developed rapidly, and the research focus of object image classification has shifted from coarse-grained image classification to fine-grained image classification. Fine-grained image classification involves identifying subclasses of a base class, such as distinguishing different species of birds or different brands of cars. Compared to coarse-grained image classification, the differences between fine-grained image categories are much more subtle, often requiring only minute local differences for accurate differentiation. Furthermore, compared to object-level classification tasks like face recognition, the differences within fine-grained image categories are even more subtle, and there are many uncertainties such as pose, occlusion, and background interference, making this task extremely challenging. This research currently mainly includes identifying different species of birds, dogs, flowers, cars, and airplanes.

[0003] Fine-grained image classification neural network models have seen widespread business demand and application scenarios in industry and daily life in recent years. Examples include the flower identification feature in "Flower Master" and the car identification feature in the mobile app "Dongchedi." In ecological protection, effectively identifying different species of organisms is a crucial prerequisite for ecological research. Therefore, if we can leverage computer vision technology to achieve low-cost, high-precision fine-grained image recognition and classification, it will be of great significance to both academia and industry.

[0004] Research indicates that existing fine-grained image classification methods can be categorized into two types: those that use only visual information and those that incorporate additional information. The former relies entirely on visual information to solve the classification problem, while the latter attempts to use additional information for classification.

[0005] Classification methods using only visual information can be broadly categorized into two types: those based on localization-classification subnetworks and those based on higher-order feature encoding. Localization-classification subnetwork-based methods detect and locate the discriminative parts of an object and establish corresponding local feature representations. Early work used part annotation as strong supervision to make the network focus on subtle differences between categories, but part annotation information is costly. Therefore, most current mainstream methods use weak supervision, i.e., only using image-level labels for classification. Higher-order feature encoding-based methods integrate features generated by neural networks at a higher order to obtain more discriminative features. However, both types of methods have their limitations: localization-classification subnetwork-based methods mostly focus on the most salient parts of an object, ignoring those less salient but discriminative parts, resulting in insufficient discriminative power in the features. Higher-order feature encoding-based methods require significant computational resources when the feature map channel dimension is high and lack sufficient interpretability.

[0006] Adding additional information for classification involves building joint feature representations by incorporating extra data (such as network data, multimodal data, etc.), where multimodal data includes sounds, textual descriptions of objects, etc. By combining rich additional information with deep neural network architectures, such methods achieve effective classification of fine-grained images. However, the limitation of these methods is that they are designed for specific prior knowledge and cannot arbitrarily apply other auxiliary information. Summary of the Invention

[0007] To address the challenges of fine-grained image classification and the limitations of attention mechanisms in this field in existing technologies, this invention proposes a fine-grained image classification method based on convolutional neural networks. It proposes a convolutional neural network that integrates channel feature re-attention modules and spatial multi-region feature attention modules to classify fine-grained images.

[0008] The technical solution of the present invention is as follows:

[0009] A fine-grained image classification method based on convolutional neural networks first constructs a classification network model by fusing a channel feature re-attention module and a spatial multi-region feature attention module. Then, a contrastive learning loss term is designed in the loss function using the idea of ​​contrastive learning. Finally, this classification network model is used to classify images acquired in real time. The specific steps include the following:

[0010] Step 1: Construct a classification network model;

[0011] The classification network model includes a feature extraction network, a channel feature re-attention module, a spatial multi-region feature attention module, and a classifier;

[0012] Step 2: Construct a training dataset and train the model;

[0013] Step 3: Acquire the image to be classified in real time, feed it into the trained classification network model, and obtain the classification result of the current image.

[0014] Furthermore, a convolutional neural network output from the last three stages is used as the feature extraction network. This feature extraction network is composed of ResNet50, ResNet101, and DenseNet161 basic convolutional networks. Each convolutional network structure consists of multiple stages, with each stage containing one convolutional layer. When an image is input into the feature extraction network, the spatial size of the feature map is halved and the number of channels doubles after each stage. The output feature maps X from the multiple stages of the feature extraction network are then processed. l As output features of the feature extraction network.

[0015] Furthermore, the channel feature re-attention module first employs average pooling and max pooling operations to integrate feature channel information, and uses the SoftMax function to obtain the weight information of each channel in the feature map; an enhanced mask matrix E is obtained according to the weight allocation, and channels with high weights are suppressed, obtaining the suppressed mask matrix S through the suppression function F(x); the input feature map X is then processed... l The output feature map is obtained by multiplying it with the enhancement mask matrix E and the suppression mask matrix S respectively. and in,

[0016] The SoftMax function is represented by the following formula:

[0017]

[0018] Among them, Z i It is the output value of each channel after passing through the SoftMax function, and C is the total number of output channels. The weight information of the desired channel is obtained through the SoftMax function.

[0019] The enhanced mask matrix E is calculated using the following formula:

[0020] E = SoftMax(AvgPool(X) l )+MaxPool((X l (2)

[0021] Where AvgPool(·) represents average pooling, and MaxPool(·) represents max pooling;

[0022] The suppression function F(x) is expressed by the following equation:

[0023]

[0024] Among them, Z maxIt is the maximum output value of the channel. ω and δ are both hyperparameters, representing the degree to which the corresponding channel is suppressed and the degree to which the channel needs to be suppressed, respectively.

[0025] Output feature map at the current stage and We obtain it from the following formula:

[0026]

[0027] in, This represents the element-wise multiplication operation;

[0028] Multiple stages After the Conv channel dimension is unified through the convolutional layer, it becomes the output of the corresponding stage. Channel unification is to ensure the balance between low-level and high-level information. The input to subsequent stages forces the network to mine potential channel features containing fine-grained knowledge.

[0029] Furthermore, the spatial multi-region feature attention module employs downsampling convolution, 1×1 convolution, the SoftMax function, and the CCMP module, where downsampling convolution is used to integrate multiple stages of features. and the feature map of the last stage of the network Maintaining a consistent spatial scale, 1×1 convolutions are used to simplify computation, while the SoftMax function and the CCMP module are used to compute multiple stages. Spatial similarity, and obtain diversity learning loss L div L div There is a negative correlation between similarity and diversity loss; by reducing diversity loss through training, multiple stages can be improved. Focus on the different distinguishable parts of an object in space;

[0030] Assume that the feature maps obtained by the last three stages of the feature extraction network after passing through the channel attention module are as follows: and Among them, C t W represents the normalized channel dimension. L-2 H represents the width of the feature map in the (L-2)th stage. L-2 W represents the height of the feature map in the (L-2)th stage. L-1 H represents the width of the feature map in the (L-1)th stage. L-1 W represents the height of the feature map in the (L-1)th stage. L H represents the width of the feature map in the Lth stage. L This represents the height of the feature map in the Lth stage;

[0031] To reduce computational cost, the feature map is preprocessed using the following formula:

[0032]

[0033] Where φ(·) represents a 1×1 convolution; Conv_block l (·) indicates downsampling convolution; l indicates the nth stage of the feature map;

[0034] After obtaining feature maps from three stages with the same spatial size and one channel, the SoftMax function is used to obtain the weight information at each spatial location, and then the feature maps are concatenated along the channel dimension to obtain the final feature map. It is input into the CCMP module, and CCMP supports X. concat Each pixel in the array responds to the peak value in the channel dimension and to X. concat The similarity value S is obtained by performing a summation and then a mean operation h(·) on the elements. i ;

[0035]

[0036] Where k represents X concat The size of the space, j represents the size of X. concat The nth channel, ε represents X concat The number of channels, through the spatial multi-region feature attention module, will yield a value S representing the similarity between feature maps at each stage. i ;

[0037] Finally, based on similarity S i Obtain diversity learning loss L div The calculation method is as follows:

[0038] L div =(1-S i ) / ε (7)

[0039] Where ε represents the output of several stages of the feature extraction network, and here it represents X. concat The number of channels.

[0040] Furthermore, the classifier uses the SoftMax classifier, which is applied in multi-class classification tasks to map the outputs of multiple neurons to the (0,1) interval.

[0041] Furthermore, the total loss function L of the classification network model total The definition is as follows:

[0042] L total =αL cls +βL div +γL con (8)

[0043] Among them, L clsL represents the cross-entropy loss. div L represents the diversity learning loss. con Let represent the contrastive learning loss, where α, β, and γ are all balancing parameters used as weights in each balancing loss function; where,

[0044] Cross-entropy loss L cls The classification loss consists of the individual classification loss of each stage and the overall classification loss formed by concatenating the features from each stage. The calculation formula is as follows:

[0045]

[0046] Where y is the ground truth label of the input image, represented by a one-hot vector; θ1 and θ2 are also balancing parameters, and the SoftMax function is used to calculate the predicted label values ​​of the neural network; cls l (·) represents a classifier. f represents the output feature of the l-th stage. l The predicted label value; cls concat (·) denotes a classifier used for overall feature representation, Z fconcat The overall feature is represented by f. concate The predicted value of the label;

[0047] Contrast learning loss L con for:

[0048]

[0049] Where N is the size of the input image batch, z i ,z j These are input images of different categories within the same batch after L2 regularization, y i ,y j Sim(z) represents the label values ​​of different categories of input images. i ,z j ) is z i ,z j The cosine similarity between the samples, where i,j represent different samples from the same batch, and η represents the similarity between inputs of different categories with a similarity greater than η, affects the loss L. con It has made a contribution.

[0050] Furthermore, the specific process of step 2 is as follows:

[0051] Step 2.1: Using the CUB_200_2011 dataset as the training dataset, the collected original images are preprocessed by horizontal flipping and center cropping to expand the data and construct the training dataset.

[0052] Step 2.2: Feed the fine-grained images of the training dataset into the classification network model, train and optimize the learnable parameters in the classification network model, so that the channel feature attention module in the model can extract the potential fine-grained knowledge in the feature map to the greatest extent, and the spatial multi-region feature attention module can greatly reduce the similarity between feature maps at different stages. When the entire model is trained to convergence, the trained classification network model is obtained.

[0053] Furthermore, the specific process of step 3 is as follows:

[0054] First, the fine-grained image to be classified is fed into a feature extraction network with stage L, and then input into a channel feature attention module to obtain a channel-enhanced feature map. Feature map of channel suppression Channel-enhanced feature maps are used as the output of the current stage of the network, while channel-suppressed feature maps are fed into subsequent stages to force the network to focus on information-poor channels that contain fine-grained knowledge. During model training, a spatial multi-region feature attention module is used to enhance the channel-enhanced feature maps output from multiple stages. The model focuses on different discriminative parts of an object in the spatial dimension; therefore, it will obtain multiple output features that are discriminative in both space and channels. Finally, the output features from multiple stages are used as the feature representation of the image. Finally, the image is classified by the SoftMax classifier.

[0055] The beneficial technical effects of this invention are as follows:

[0056] This invention significantly improves upon the limitations of attention mechanisms and convolutional neural network-based methods in fine-grained image classification. Through a multi-stage feature extraction network, it enhances the classification network's ability to aggregate feature information, incorporating both low-level and high-level semantic information, thus increasing the robustness of the extracted features. The channel feature re-attention module effectively helps the classification network extract previously overlooked but valuable channel features for fine-grained classification, resulting in a more comprehensive feature representation. The spatial multi-region feature attention module ensures that the features output from multiple stages of the classification network focus on different discriminative parts of the object in the spatial dimension, improving the discriminativeness of the final feature representation. By incorporating a loss term based on contrastive learning, it treats different categories of fine-grained images differently, increasing inter-class differences. The contrastive learning loss term integrates the idea of ​​contrastive learning, setting different types of training images in the same input batch as negative samples and the same type as positive samples. By adjusting the loss function, it narrows the distance between positive samples and widens the distance between negative samples, thereby further optimizing the classification network's performance during training. Attached Figure Description

[0057] Figure 1 This is a flowchart of the fine-grained image classification method based on convolutional neural networks of the present invention;

[0058] Figure 2 This is a schematic diagram of the overall structure of the classification network model in this invention;

[0059] Figure 3 This is a schematic diagram of the channel feature re-attention module in the classification network model of this invention;

[0060] Figure 4 This is a schematic diagram of the spatial multi-region feature attention module in the classification network model of this invention. Detailed Implementation

[0061] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0062] Research has found that among numerous fine-grained image classification methods, convolutional neural networks employing a fusion of channel feature re-attention modules and spatial multi-region feature attention modules are a relatively reliable approach. This is a weakly supervised method, and using a multi-stage convolutional network as the feature extraction network yields more comprehensive and richer features. This is because multi-stage features contain both low-level information (color, edge connectivity, etc.) and high-level semantic information. Low-level information remains unchanged when the object's pose and background change, reducing intra-class variance. Although classification methods based on deep learning and attention mechanisms have improved the performance of fine-grained image classification to some extent, some shortcomings remain. For fine-grained image classification networks, in addition to extracting saliency and easily distinguishable features, it is also necessary to help the neural network learn more knowledge helpful for fine-grained classification in the channel and spatial dimensions of object features. This can be achieved by using a channel feature re-attention module to force the network to mine knowledge from information-poor channel features, and by using a spatial multi-region feature attention module to allow multi-stage features to focus on different discriminative parts of the object. Ultimately, this results in more discriminative feature representations in both the channel and spatial dimensions.

[0063] Therefore, this invention proposes a fine-grained image classification method based on convolutional neural networks. A classification network model is constructed by fusing a channel feature re-attention module and a spatial multi-region feature attention module. A contrastive learning loss term is designed in the loss function using the idea of ​​contrastive learning. Finally, this classification network model is used to classify images acquired in real time. Figure 1 and Figure 2 As shown, the specific steps include the following:

[0064] Step 1: Construct a classification network model;

[0065] The classification network model includes a feature extraction network, a channel feature re-attention module, a spatial multi-region feature attention module, and a classifier.

[0066] The feature extraction network is composed of basic convolutional networks such as ResNet50, ResNet101, and DenseNet161. These convolutional networks have similar structures, each consisting of multiple stages, with each stage containing a convolutional layer. When an image is input into the feature extraction network, the spatial size of the feature map is halved and the number of channels doubles after each stage. The output feature map X from the multiple stages of the feature extraction network is then processed. l As the output feature of the feature extraction network, and referred to as a multi-stage convolutional neural network, this invention uses the convolutional neural network with the output of the last three stages as the feature extraction network.

[0067] like Figure 3 As shown, the channel feature re-attention module first uses average pooling and max pooling operations to integrate feature channel information, and uses the SoftMax function to obtain the weight information of each channel in the feature map; an enhanced mask matrix E is obtained according to the weight allocation, and channels with high weights are suppressed, and a suppressed mask matrix S is obtained through the suppression function F(x). The input feature map X is then processed. l The output feature map is obtained by multiplying it with the enhancement mask matrix E and the suppression mask matrix S respectively. and in,

[0068] The SoftMax function can be represented by the following formula:

[0069]

[0070] Among them, Z i C is the output value of each channel after passing through the SoftMax function, and C is the total number of output channels. The weight information of the desired channel can be obtained through the SoftMax function.

[0071] The enhanced mask matrix E can be calculated using the following formula:

[0072] E = SoftMax(AvgPool(X) l )+MaxPool((X l (2)

[0073] Where AvgPool(·) represents average pooling, and MaxPool(·) represents max pooling;

[0074] The suppression function F(x) can be expressed by the following equation:

[0075]

[0076] Among them, Z i Z is the output value of each channel after passing through the SoftMax function. max ω is the maximum output value of the channel. ω and δ are both hyperparameters, representing the degree to which the corresponding channel is suppressed and the degree to which the channel needs to be suppressed, respectively.

[0077] Output feature map at the current stage and It can be obtained from the following formula:

[0078]

[0079] in, This indicates an element-wise multiplication operation.

[0080] Multiple stages After the Conv channel dimensions of the convolutional layer are unified, it becomes the output of the corresponding stage. Channel unification is to ensure the balance between low-level and high-level information. The input to subsequent stages forces the network to mine potential channel features containing fine-grained knowledge.

[0081] like Figure 4 As shown, the spatial multi-region feature attention module employs downsampling convolution, 1×1 convolution, the SoftMax function, and the CCMP (Cross-channel max pooling) module. The downsampling convolution is used to integrate features from multiple stages. and the feature map of the last stage of the network Maintaining a consistent spatial scale, 1×1 convolutions are used to simplify computation, while the SoftMax function and the CCMP module are used to compute multiple stages. Spatial similarity, and obtain diversity learning loss L div L div There is a negative correlation between similarity and diversity loss; by reducing diversity loss through training, multiple stages can be improved. Focus on the different distinguishable parts of an object in space;

[0082] Assume that the feature maps obtained by the last three stages of the feature extraction network after passing through the channel attention module are as follows: and Among them, C t W represents the normalized channel dimension, which is equal to 1 in this invention. L-2 H represents the width of the feature map in the (L-2)th stage. L-2 W represents the height of the feature map in the (L-2)th stage. L-1 H represents the width of the feature map in the (L-1)th stage. L-1 W represents the height of the feature map in the (L-1)th stage. LH represents the width of the feature map in the Lth stage. L This represents the height of the feature map in the Lth stage;

[0083] To reduce computational cost, the feature map is preprocessed using the following formula:

[0084]

[0085] Where φ(·) represents a 1×1 convolution; Conv_block l (·) indicates downsampling convolution; l indicates the nth stage of the feature map.

[0086] This yields feature maps from three stages with the same spatial size and one channel. To explore the similarity in spatial size among the feature maps from the three stages, the SoftMax function is used to obtain the weight information at each spatial location, and then the maps are concatenated along the channel dimension to obtain the final feature map. It is fed into the CCMP module, which is cross-channel max pooling, and it tends to focus on X. concat Each pixel in the array responds to the peak value in the channel dimension and to X. concat The similarity value S is obtained by performing a summation and then a mean operation h(·) on the elements. i ;

[0087]

[0088] Where k represents X concat The size of the space, j represents the size of X. concat The nth channel, ε represents X concat The number of channels, through the spatial multi-region feature attention module, will yield a value S representing the similarity between feature maps at each stage. i S i The higher the value of S, the higher the similarity between feature maps. To allow the classification model to focus on multiple different parts of an object, the similarity between feature maps is reduced during training, i.e., S is reduced. i .

[0089] Finally, based on similarity S i Obtain diversity learning loss L div The calculation method is as follows:

[0090] L div =(1-S i ) / ε (7)

[0091] Where ε represents the output of several stages of the feature extraction network, and here it represents X. concat The number of channels is 3 in this invention.

[0092] The classifier used is the SoftMax classifier, which is used in multi-class classification tasks. It can map the outputs of multiple neurons to the (0,1) interval, which can be understood as probabilities, and thus perform multi-class classification.

[0093] In addition, the total loss function L of the classification network model total The definition is as follows:

[0094] L total =αL cls +βL div +γL con (8)

[0095] Among them, L cls L represents the cross-entropy loss. div L represents the diversity learning loss. con Let represent the contrastive learning loss, and α, β, γ represent the balancing parameters used as weights for each balancing loss function; where,

[0096] Cross-entropy loss L cls The classification loss consists of the individual classification loss of each stage and the overall classification loss formed by concatenating the features from each stage. The calculation formula is as follows:

[0097]

[0098] Here, y is the ground truth label of the input image, represented by a one-hot vector. θ1 and θ2 are also balancing parameters, and the SoftMax function is used to calculate the predicted label values ​​of the neural network. cls l (·) represents a classifier. f represents the output feature of the l-th stage. l The predicted label value. cls concat (·) denotes a classifier used for overall feature representation, Z fconcat The overall feature is represented by f. concate The predicted value of the label.

[0099] Diversity learning loss L div The calculation formula is formula (6):

[0100] Contrast learning loss L con for:

[0101]

[0102] Where N is the size of the input image batch, z i ,z j These are input images of different categories within the same batch after L2 regularization, y i ,y jSim(z) represents the label values ​​of different categories of input images. i ,z j ) is z i ,z j The cosine similarity between the samples, where i,j represent different samples from the same batch, and η represents the similarity between inputs of different categories with a similarity greater than η, affects the loss L. con It has made a contribution.

[0103] Step 2: Construct the training dataset and train the model; the specific process is as follows:

[0104] Step 2.1: Using the CUB_200_2011 dataset as the training dataset, the collected original images are preprocessed by methods such as horizontal flipping and center cropping to expand the data and construct the training dataset;

[0105] The CUB_200_2011 dataset, proposed by Caltech in 2010, is a fine-grained dataset and is currently the benchmark image dataset for fine-grained classification and recognition research. It contains 11,788 bird images, including 200 bird subclasses, with 5,994 images in the training dataset and 5,794 images in the test set. Each image is accompanied by image class labeling information.

[0106] Step 2.2: Feed the fine-grained images of the training dataset into the classification network model, train and optimize the learnable parameters in the classification network model, so that the channel feature attention module in the model can extract the potential fine-grained knowledge in the feature map to the greatest extent, and the spatial multi-region feature attention module can greatly reduce the similarity between feature maps at different stages. When the entire model is trained to convergence, the trained classification network model is obtained.

[0107] Step 3: Acquire the image to be classified in real time and feed it into the trained classification network model to obtain the classification result of the current image. The specific process is as follows:

[0108] First, the fine-grained image to be classified is fed into a feature extraction network with stage L, and then input into a channel feature attention module to obtain a channel-enhanced feature map. Feature map of channel suppression Channel-enhanced feature maps are used as the output of the current stage of the network, while channel-suppressed feature maps are fed into subsequent stages to force the network to focus on information-poor channels that contain fine-grained knowledge. During model training, a spatial multi-region feature attention module is used to enhance the channel-enhanced feature maps output from multiple stages. The model focuses on different discriminative parts of an object in the spatial dimension; therefore, it will obtain multiple output features that are discriminative in both space and channels. Finally, the output features from multiple stages are used as the feature representation of the image. Finally, the image is classified by the SoftMax classifier.

[0109] This invention proposes a fine-grained image classification method based on a convolutional neural network that integrates a channel feature re-attention module and a spatial multi-region feature attention module. This method combines convolutional neural networks from deep learning with an improved attention module for fine-grained image classification. The method described in this invention significantly improves upon the shortcomings of the attention mechanism in this task, enhancing the feature extraction capability of the basic convolutional network. In the classification network model, the proposed channel feature re-attention module improves the utilization rate of features in the network with almost no increase in the original network's learning parameters. It also better learns the fine-grained knowledge contained in the channel features that are potentially helpful for fine-grained classification, while controlling overfitting to smaller training sets (such as the CUB_200_2011 bird dataset used in this invention). The spatial multi-region feature attention module allows the feature maps output by multiple stages of the classification network to focus on different discriminative parts of the object spatially, rather than all focusing on the most salient parts. The contrastive learning loss term designed in the loss function integrates the idea of ​​contrastive learning, improving the classification performance of the network model. This invention addresses the problem that deep networks cannot fully utilize context when extracting features in fine-grained image classification tasks, and that attention mechanisms only focus on the most salient channel and spatial features of objects when applied.

[0110] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A fine-grained image classification method based on convolutional neural networks, characterized in that, First, a classification network model was constructed by fusing channel feature attention modules and spatial multi-region feature attention modules. Then, a contrastive learning loss term was designed in the loss function using the idea of ​​contrastive learning. Finally, this classification network model was used to classify images acquired in real time. The specific steps include the following: Step 1: Construct a classification network model; The classification network model includes a feature extraction network, a channel feature re-attention module, a spatial multi-region feature attention module, and a classifier; The channel feature re-attention module first uses average pooling and max pooling operations to integrate feature channel information, and uses the SoftMax function to obtain the weight information of each channel in the feature map; it then obtains the enhanced mask matrix E according to the weight allocation, and suppresses the channels with high weights, obtaining the suppressed mask matrix S through the suppression function F(x); Input feature map X l The output feature map is obtained by multiplying it with the enhancement mask matrix E and the suppression mask matrix S respectively. and The spatial multi-region feature attention module employs downsampling convolution, 1×1 convolution, the SoftMax function, and the CCMP module, where downsampling convolution is used to integrate multiple stages... and the feature map of the last stage of the network Maintaining a consistent spatial scale, 1×1 convolutions are used to simplify computation, while the SoftMax function and the CCMP module are used to compute multiple stages. Spatial similarity, and obtain diversity learning loss L div L div There is a negative correlation between similarity and diversity loss; by reducing diversity loss through training, multiple stages can be improved. Spatially, it focuses on different discriminative parts of an object; the CCMP module analyzes the feature map X. concat Each pixel in the array responds to the peak value in the channel dimension and to X. concat The similarity value S is obtained by performing a summation and then a mean operation h(·) on the elements. i ; Step 2: Construct a training dataset and train the model; Step 3: Acquire the image to be classified in real time, feed it into the trained classification network model, and obtain the classification result of the current image.

2. The fine-grained image classification method based on convolutional neural networks according to claim 1, characterized in that, A convolutional neural network output from the last three stages is used as the feature extraction network. This network consists of ResNet50, ResNet101, and DenseNet161 basic convolutional networks, each with multiple stages, each containing one convolutional layer. When an image is input into the feature extraction network, the spatial size of the feature map is halved and the number of channels doubles after each stage. The output feature map X from the multiple stages of the feature extraction network is then processed. l As output features of the feature extraction network.

3. The fine-grained image classification method based on convolutional neural networks according to claim 1, characterized in that, In step 1, the SoftMax function is represented by the following formula: Among them, Z i It is the output value of each channel after passing through the SoftMax function, and C is the total number of output channels. The weight information of the desired channel is obtained through the SoftMax function. The enhanced mask matrix E is calculated using the following formula: E=SoftMax(AvgPool(X l )+MaxPool((X l ))) (2) Where AvgPool(·) represents average pooling, and MaxPool(·) represents max pooling; The suppression function F(x) is expressed by the following equation: Among them, Z max It is the maximum output value of the channel. ω and δ are both hyperparameters, representing the degree to which the corresponding channel is suppressed and the degree to which the channel needs to be suppressed, respectively. Output feature map at the current stage and We obtain it from the following formula: in, This represents the element-wise multiplication operation; Multiple stages After the Conv channel dimension is unified through the convolutional layer, it becomes the output of the corresponding stage. Channel unification is to ensure the balance between low-level and high-level information. The input to subsequent stages forces the network to mine potential channel features containing fine-grained knowledge.

4. The fine-grained image classification method based on convolutional neural networks according to claim 1, characterized in that, In step 1, it is assumed that the feature maps obtained by the last three stages of the feature extraction network after passing through the channel re-attention module are as follows: and Among them, C t W represents the normalized channel dimension. L-2 H represents the width of the feature map in the (L-2)th stage. L-2 W represents the height of the feature map in the (L-2)th stage. L-1 H represents the width of the feature map in the (L-1)th stage. L-1 W represents the height of the feature map in the (L-1)th stage. L H represents the width of the feature map in the Lth stage. L This represents the height of the feature map in the Lth stage; To reduce computational cost, the feature map is preprocessed using the following formula: Where φ(·) represents a 1×1 convolution; Conv_block l (·) indicates downsampling convolution; l indicates the nth stage of the feature map; After obtaining feature maps from three stages with the same spatial size and one channel, the SoftMax function is used to obtain the weight information at each spatial location, and then the feature maps are concatenated along the channel dimension to obtain the final feature map. Input it into the CCMP module. The calculation formula for the CCMP model is as follows: Where k represents X concat The size of the space, j represents the size of X. concat The nth channel, ε represents X concat The number of channels, through the spatial multi-region feature attention module, will yield a value S representing the similarity between feature maps at each stage. i ; Finally, based on similarity S i Obtain diversity learning loss L div The calculation method is as follows: L div =(1-S i ) / e (7) Where ε represents the output of several stages of the feature extraction network, and here it represents X. concat The number of channels.

5. The fine-grained image classification method based on convolutional neural networks according to claim 1, characterized in that, The classifier used is the SoftMax classifier, which is applied in multi-class classification tasks to map the outputs of multiple neurons to the (0,1) interval.

6. The fine-grained image classification method based on convolutional neural networks according to claim 1, characterized in that, The total loss function L of the classification network model total The definition is as follows: L total =αL cls +βL div +γL con (8) Among them, L cls L represents the cross-entropy loss. div L represents the diversity learning loss. con Let represent the contrastive learning loss, where α, β, and γ are all balancing parameters used as weights in each balancing loss function; where, Cross-entropy loss L cls The classification loss consists of the individual classification loss of each stage and the overall classification loss formed by concatenating the features from each stage. The calculation formula is as follows: Where y is the ground truth label of the input image, represented by a one-hot vector; θ1 and θ2 are also balancing parameters, and the SoftMax function is used to calculate the predicted label values ​​of the neural network; cls l (·) represents a classifier. f represents the output feature of the l-th stage. l The predicted label value; cls concat (·) denotes a classifier used for overall feature representation, Z fconcat The overall feature is represented by f. concate The predicted value of the label; Contrast learning loss L con for: Where N is the size of the input image batch, z i ,z j These are input images of different categories within the same batch after L2 regularization, y i ,y j Sim(z) represents the label values ​​of different categories of input images. i ,z j ) is z i ,z j The cosine similarity between the samples, where i,j represent different samples from the same batch, and η represents the similarity between inputs of different categories with a similarity greater than η, affects the loss L. con It has made a contribution.

7. The fine-grained image classification method based on convolutional neural networks according to claim 1, characterized in that, The specific process of step 2 is as follows: Step 2.1: Using the CUB_200_2011 dataset as the training dataset, the collected original images are preprocessed by horizontal flipping and center cropping to expand the data and construct the training dataset. Step 2.2: Feed the fine-grained images of the training dataset into the classification network model, train and optimize the learnable parameters in the classification network model, so that the channel feature attention module in the model can extract the potential fine-grained knowledge in the feature map to the greatest extent, and the spatial multi-region feature attention module can greatly reduce the similarity between feature maps at different stages. When the entire model is trained to convergence, the trained classification network model is obtained.

8. The fine-grained image classification method based on convolutional neural networks according to claim 1, characterized in that, The specific process of step 3 is as follows: First, the fine-grained image to be classified is fed into a feature extraction network with stage L, and then input into a channel feature attention module to obtain a channel-enhanced feature map. Feature map of channel suppression Channel-enhanced feature maps are used as the output of the current stage of the network, while channel-suppressed feature maps are fed into subsequent stages to force the network to focus on information-poor channels that contain fine-grained knowledge. During model training, a spatial multi-region feature attention module is used to enhance the channel-enhanced feature maps output from multiple stages. The model focuses on different discriminative parts of an object in the spatial dimension; therefore, it will obtain multiple output features that are discriminative in both space and channels. Finally, the output features from multiple stages are used as the feature representation of the image. Finally, the image is classified by the SoftMax classifier.

Images