An image classification method based on Resnet50 combined with attention mechanism and feature pyramid

Abstract

The present invention discloses an image classification method based on Resnet50 combined with an attention mechanism and a feature pyramid in the field of image processing technology, comprising the following steps: S1, collecting input images and performing data preprocessing on the images; S2, importing the preprocessed images into a classification model, and outputting feature maps of three different convolutional layers; S3, performing classification prediction on each feature map separately, taking a classifier with the maximum credibility, and outputting a final predicted classification result. The present invention has the advantages of high image classification accuracy, high extraction quality, and the like.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image classification method based on Resnet50 combined with an attention mechanism and a feature pyramid. Background Art

[0002] As a classic backbone framework for deep learning, Resnet proposes a residual block, which solves the problem of deep networks being difficult to train, gradient vanishing, and gradient exploding, greatly enhancing the ability of neural networks to extract features. Resnet has achieved remarkable results in improving the training speed of neural networks, but its specific application in computer vision classification, positioning, detection, and segmentation needs to be improved.

[0003] However, the inventors of this application found that the above technology has at least the following technical problems in the process of implementing the technical solution of the invention in the embodiment of this application:

[0004] In the field of image classification, since the scales of the classification targets in each image vary greatly, the inconsistent target sizes increase the difficulty of image classification, resulting in unsatisfactory final classification results.

[0005] For example, the dataset cifar-10 is used for classification tasks. The dataset includes 60,000 32x32 color images, including 50,000 training sets and 10,000 test sets. Cifar-10 is labeled into 10 categories, with 6,000 images in each category. The 10 categories are airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. There is no overlap, that is, airplane only includes airplanes, and automobile only includes small cars. There will be no two types of objects in the same photo. CIFAR-10 contains real objects in the real world. Not only is the noise very large and the resolution low, but the proportions and features of the objects are also different, which brings great difficulties to recognition.

[0006] Based on this, the present invention designs an image classification method based on Resnet50 combined with attention mechanism and feature pyramid to solve the above problems. Summary of the invention

[0007] In order to solve the technical problems mentioned in the current background technology, the purpose of the present invention is to provide an image classification method based on Resnet50 combined with an attention mechanism and a feature pyramid.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] An image classification method based on Resnet50 combined with attention mechanism and feature pyramid includes the following steps:

[0010] S1, collect input pictures and perform data preprocessing on the images;

[0011] S2, import the preprocessed image into the classification model and output the feature maps of three different convolutional layers;

[0012] S3. Perform classification prediction on each feature map separately, select the classifier with the maximum credibility, and output the final prediction classification result.

[0013] Preferably, the data preprocessing includes:

[0014] Process the input image format (c, h, w) to (c, 7h, 7w);

[0015] Among them, c is the number of image channels, h is the image height, and w is the image height.

[0016] Preferably, the processing of the classification model includes:

[0017] Perform focus module processing on the preprocessed image to extract different features;

[0018] Meaningful features are obtained through processing by the CBAM attention mechanism module;

[0019] Extract image features and output feature layers at the bottom-top layers;

[0020] Perform channel reduction operation on the feature layer after convolution to obtain an intermediate feature layer with a uniform number of channels;

[0021] Corresponding to the bottom-top layer fusion output feature layer.

[0022] Preferably, the focus module processing includes:

[0023] Slice the input image and concatenate it into a new image, where the number of channels of the new image is half of the input image and the size of the new image is four times that of the input image;

[0024] Convolution outputs a feature map and switches the information on the plane to the channel dimension, extracting different features through convolution.

[0025] Preferably, the CBAM attention mechanism module processing includes:

[0026] Spatial attention focuses on the target area in the feature map, obtains detailed information of the target, and suppresses other useless information.

[0027] Preferably, the fusion output feature layer includes:

[0028] The low-resolution high-level abstract semantic information in the upper-level feature map is fused into the low-level high-resolution low-semantic features through add.

[0029] Preferably, the classification prediction includes:

[0030] The three different feature layers are used as output results for the fully connected layer fc operation;

[0031] After fully connecting each feature layer, classify it and output a probability value for each category;

[0032] The classification result with the largest probability value is taken as the final output prediction value.

[0033] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0034] 1. The present invention improves the feature extraction effect of the Resnet50 model by combining the attention module CBAM and the feature pyramid FPN structure;

[0035] 2. The present invention uses an optimization algorithm based on three classifiers of different feature layers of FPN to make full use of the feature information of different layers after fusion, thereby improving the accuracy of classification;

[0036] In summary, the present invention has the advantages of high image classification accuracy and high extraction quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0038] Figure 1 is a processing flow chart of the image classification model of the present invention;

[0039] Figure 2 These are Figures P4, P5 and P6 of the present invention. DETAILED DESCRIPTION

[0040] The following is a description of the implementation of the present invention by means of specific embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification.

[0041] Embodiment 1

[0042] The present invention provides a technical solution: an image classification method based on Resnet50 combined with an attention mechanism and a feature pyramid, comprising the following steps:

[0043] S1, collect input pictures and perform data preprocessing on the images;

[0044] S2, import the preprocessed image into the classification model and output the feature maps of three different convolutional layers;

[0045] S3. Perform classification prediction on each feature map separately, select the classifier with the maximum credibility, and output the final prediction classification result.

[0046] Through the above steps, it is not difficult to find that in the image classification process of the present invention, the classification model is an image processing model based on Resnet50, attention mechanism module and FPN module. Through three classifiers based on FPN, the classifier with the highest classification accuracy is finally taken as the final classification result. Combined with the lightweight versatility of the attention module CBAM, the feature extraction capability of the overall network framework is enhanced. Finally, on the cifar10 public data set, the present invention can improve the accuracy of Resnet50 by about 1.28% in terms of classification accuracy through the improved Resnet50-like network structure.

[0047] It should be noted that, in the present invention, CBAM is an attention mechanism and FPN is a feature pyramid.

[0048] In order to better realize the input of image data to the focus module, the data preprocessing includes:

[0049] Process the input image format (c, h, w) to (c, 7h, 7w);

[0050] Among them, c is the number of image channels, h is the image height, and w is the image height.

[0051] In this embodiment, it should be supplemented that, after each convolution layer, a normalization layer (BN) and an activation layer (ReLU) are performed; for example, when the input image data format is (3, 32, 32), it becomes (3, 224, 224) after data preprocessing and transformation.

[0052] In order to achieve the output of three different convolutional layer feature maps, the processing of the classification model includes:

[0053] Perform focus module processing on the preprocessed image to extract different features;

[0054] Meaningful features are obtained through processing by the CBAM attention mechanism module;

[0055] Extract image features and output feature layers at the bottom-top layers;

[0056] Perform channel reduction operation on the feature layer after convolution to obtain an intermediate feature layer with a uniform number of channels;

[0057] Corresponding to the bottom-top layer fusion output feature layer.

[0058] Furthermore, the focus module processing includes:

[0059] Slice the input image and concatenate it into a new image, where the number of channels of the new image is half of the input image and the size of the new image is four times that of the input image;

[0060] Convolution outputs a feature map and switches the information on the plane to the channel dimension, extracting different features through convolution.

[0061] In this embodiment, the input image is first sliced, then connected together through the concat operation to form an image with a size of half the input image and a channel number of four times the input image, and finally convolution is performed to output a feature map. The information on the plane is converted to the channel dimension, and then different features are extracted by convolution. The purpose of using the Focus layer is downsampling. The Focus layer can effectively reduce the information loss caused by downsampling and reduce the amount of calculation.

[0062] In order to better complete feature extraction, the CBAM attention mechanism module processing includes:

[0063] Spatial attention focuses on the target area in the feature map, obtains detailed information of the target, and suppresses other useless information.

[0064] In this embodiment, since each feature map is equivalent to capturing a feature in the original image, channel attention helps to screen out meaningful features, that is, telling CNN which part of the original image has meaningful features. Since a pixel in the feature map represents a certain feature of a certain area in the original image, spatial attention is equivalent to telling the network which area of ​​the original image should pay attention to the features, and obtain the target area that needs to be focused on, so as to obtain more detailed information of the target that needs to be focused on, and suppress other useless information.

[0065] In order to achieve the fusion of the output feature layer, the fused output feature layer includes:

[0066] The low-resolution high-level abstract semantic information in the upper-level feature map is fused into the low-level high-resolution low-semantic features through add.

[0067] In this embodiment, the semantic information of the feature map is retained to the greatest extent during the upsampling process (which is beneficial for classification), and then added and fused with the corresponding feature map with rich spatial information (high resolution, which is beneficial for positioning) during the bottom-up process, thereby obtaining a feature map with both good spatial information and strong semantic information.

[0068] It should be noted that the present invention also provides a specific implementation process of an image classification method based on Resnet50 combined with an attention mechanism and a feature pyramid, such as Figure 1 As shown, the following steps are included:

[0069] Step 1: The input image data format is (3, 32, 32), which is transformed into (3, 224, 224) after data preprocessing. Then, it passes through the focus module and draws on the idea of ​​yolov5 to perform downsampling, which effectively reduces the information loss caused by downsampling and reduces the amount of calculation. The data format is now (12, 112, 112);

[0070] Step 2: After passing through the CBAM attention mechanism module, the data format remains unchanged; since each feature map is equivalent to capturing a feature in the original image, channel attention helps to screen out meaningful features, that is, to tell CNN which part of the original image features are meaningful, obtain the target area that needs to be focused on, so as to obtain more detailed information of the target that needs to be focused on, and suppress other useless information;

[0071] Step 3: After convolution of layer 1 of Resnet50, the output is (256, 56, 56); after convolution of layer 2, the output is B3 (512, 28, 28); after convolution of layer 3, the output is B2 (1024, 14, 14); after convolution of layer 4, the output is B1 (2048, 7, 7);

[0072] Step 4: After convolution, the number of image channels of B1, B2, and B3 is reduced to 512 channels, LatB1 (512, 7, 7), LatB2 (512, 14, 14), LatB3 (512, 28, 28);

[0073] Step 5: In the up-bottom process of FPN, the low-resolution high-level abstract semantic information in the upper feature map is added to the low-level high-resolution low-semantic features to achieve information complementarity. P6 = LatB1, P5 = add(upsample(P6), LatB2), P4 = add(upsample(P5), LatB3), at this time P6 (512, 7, 7), P5 (512, 14, 14), P4 (512, 28, 28), P4, P5, P6 are the fusion output feature layers of the top-bottom layers of FPN, where the upsample operation adopts the nearest neighbor interpolation method, which can retain the semantic information of the feature map to the greatest extent during the upsampling process (which is beneficial for classification), and then add and fuse with the corresponding feature map with rich spatial information (high resolution, which is beneficial for positioning) in the bottom-up process, so as to obtain a feature map with both good spatial information and strong semantic information;

[0074] Step 6: Generally, there is only one image classifier, and the classification prediction is performed after the last feature layer of the network model is fully connected. After passing through the fully connected layers after P6, P5, and P4, three classifiers are set to make full use of the sensitivity of the feature maps of different layers after fusion to the classification targets of different scales, and the maximum value of the output classification probability values ​​of the three classifiers is output as the final prediction. The specific example principle is: the probability of the classifier 1 of the P6 layer outputting birds is 0.56, the probability of the classifier 2 of the P5 layer outputting cats is 0.63, and the probability of the classifier 3 of the P4 layer outputting birds is 0.50, then the final output result identifies the image classification result as a cat. Through this optimization, the model is encouraged to find the global optimum more quickly during the training process, and will not repeatedly search for the local optimum because only the last layer of classifiers outputs the results.

[0075] It should be noted that FC1, FC2, and FC3 represent three different feature layers, which are the output results of the fully connected layer fc operation;

[0076] C1, C2, and C3 represent three different feature layers that are fully connected and then classified, outputting a probability value for each category;

[0077] Max(C1, C2, C3) is a classification algorithm designed by the present invention, which obtains three fully connected layers from three different feature layers for comparison; utilizes the fused features of each layer to fully identify the sizes of classification targets of different scales, and takes the classification result with the largest probability value as the final output prediction value.

[0078] like Figure 2As shown in Figure 2, the P4, P5, and P6 feature layers of FPN are visualized by using Grad-CAM class activation. The normal classifier is classified after the last layer of P6 fully connected layer, but Figure 2 As shown in the figure, the deer image in the red frame in the second row and the first column has a zero contribution to classification in the final feature extraction stage, while the P4 and P5 layers retain certain image information. Similarly, the boat image in the yellow frame has a poor feature effect in the final layer, but has certain classification information in the P4 and P5 layers.

[0079] For example: by using the network model designed by the present invention, on the cifar10 data set, the training set is 50,000 pictures, the test set is 10,000 pictures, a total of 120 epochs are trained, Batchsize=20, learning-rate=0.1, and the learning rate is reduced by 10 times at 30, 60, and 90 epochs; by comparing with the top1 accuracy of 90.39% of the Resnet50 model, the top1 accuracy of this model reaches 91.67%, which improves the accuracy by 1.28%, proving the rationality of the design of this model and helping to improve the classification accuracy; therefore, this method can be extended to other image classification fields to achieve wider applications.

[0080] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. An image classification method based on Resnet50 combined with attention mechanism and feature pyramid, characterized in that: The following steps are involved: S1, collect input pictures and perform data preprocessing on the images; S2, import the preprocessed image into the classification model and output the feature maps of three different convolutional layers; S3, perform classification prediction on each feature map, select the classifier with the maximum credibility, and output the final prediction classification result; The processing of the classification model includes: Perform focus module processing on the preprocessed image to extract different features; Meaningful features are obtained through processing by the CBAM attention mechanism module; Extract image features and output feature layers at the bottom-top layers; Perform channel reduction operation on the feature layer after convolution to obtain an intermediate feature layer with a uniform number of channels; Corresponding to the bottom-top layer fusion output feature layer; The focus module processing includes: Slice the input image and concatenate it into a new image, where the number of channels of the new image is half of the input image and the size of the new image is four times that of the input image; Convolution outputs a feature map and switches the information on the plane to the channel dimension, extracting different features through convolution.

2. According to claim 1, an image classification method based on Resnet50 combined with attention mechanism and feature pyramid is characterized in that: The data preprocessing includes: Process input image format (c, h, w) to (c, 7h, 7w); Among them, c is the number of image channels, h is the image height, and w is the image height.

3. According to claim 1, an image classification method based on Resnet50 combined with attention mechanism and feature pyramid is characterized in that: The CBAM attention mechanism module processing includes: Spatial attention focuses on the target area in the feature map, obtains detailed information of the target, and suppresses other useless information.

4. According to claim 1, an image classification method based on Resnet50 combined with attention mechanism and feature pyramid is characterized in that: The fusion output feature layer includes: The low-resolution high-level abstract semantic information in the upper-level feature map is fused into the low-level high-resolution low-semantic features through add.

5. The image classification method based on Resnet50 combined with attention mechanism and feature pyramid according to claim 1, characterized in that: The classification prediction includes: The three different feature layers are used as output results for the fully connected layer fc operation; After fully connecting each feature layer, classify it and output a probability value for each category; Take the classification result with the largest probability value.

Images