An image classification method combining FCN network and Resnet network

Abstract

The present invention relates to an image classification method combining an FCN network and a Resnet network, the method comprising obtaining an image to be classified; inputting the obtained image to be classified into a trained classification model for classification to obtain a classification result, the classification model is formed by a FCN network module and a Resnet network module, wherein the FCN network module comprises an input layer, an intermediate layer and an upsampling layer connected in sequence, the Resnet network module comprises an intermediate layer and an output layer, and the output of the upsampling layer of the FCN network module and the input of the FCN network module are multiplied to obtain a result as the input of the intermediate layer of the Resnet network module. In the present invention, the multiplication operation is performed so that each pixel of the input image is added with an attention mechanism, thereby solving the problem that the classification accuracy of the Resnet classification network in the prior art is not high enough.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and particularly relates to an image classification method combining an FCN network and a ResNet network. Background Art

[0002] Deep learning technology is booming and has become a core technology in the field of artificial intelligence. Deep convolutional neural networks are key algorithms in deep learning and are widely used in computer vision, particularly in image classification, segmentation, recognition, and localization. They have achieved tremendous success, repeatedly winning international competitions in image processing and analysis of natural scene images and in various professional fields (such as medicine and remote sensing), and have essentially completely replaced traditional algorithms. Image classification is a classic problem in computer vision, and a variety of classification networks have been proposed, such as VGG, GoogLeNet, and ResNet. Compared with models such as VGG and GoogLeNet, the ResNet classification network has relatively better classification accuracy, but its classification accuracy still needs improvement. Summary of the Invention

[0003] The present invention provides an image classification method combining an FCN network and a ResNet network, so as to solve the problem that the classification accuracy of the ResNet classification network in the prior art is not high enough.

[0004] To solve the above technical problems, the present invention provides an image classification method combining an FCN network and a ResNet network, which includes:

[0005] 1) Obtain the image to be classified;

[0006] 2) Inputting the obtained image to be classified into a trained classification model for classification to obtain a classification result, wherein the classification model is jointly formed by an FCN network module and a Resnet network module, wherein the FCN network module includes an input layer, an intermediate layer, and an upsampling layer connected in sequence, and the Resnet network module includes an intermediate layer and an output layer connected in sequence, and a result obtained by multiplying the output of the upsampling layer of the FCN network module and the input of the FCN network module is used as the input of the intermediate layer of the Resnet network module; using the image data set and the corresponding classification result to train the classification model to obtain a trained classification model.

[0007] The beneficial effect of the above technical solution is: using a trained classification model to classify the image to be classified, wherein the classification model in this application uses an FCN network model with the output layer removed and a Resnet network model with the input layer removed, and multiplies the output of the upsampling layer of the FCN network model with the input of the FCN network model, and outputs the product result to the middle layer of the Resnet network model. In this case, the multiplication operation causes each pixel of the input image of the FCN network model to add an attention mechanism, and the image with the attention mechanism is used as the input of the middle layer of the Resnet network model, thereby improving the classification accuracy of the classification model. In this way, it can solve the problem of insufficient classification accuracy of the Resnet classification network in the prior art.

[0008] Furthermore, in order to obtain a more accurate classification model, the present invention provides an image classification method combining an FCN network and a Resnet network, further comprising: the number of output channels of the upsampling layer is equal to the number of input channels of the FCN network module.

[0009] Furthermore, in order to obtain a more accurate classification model, the present invention provides an image classification method combining an FCN network and a Resnet network, and further includes an FCN network module improved based on an FCN-32s network model.

[0010] Furthermore, in order to obtain a more accurate classification model, the present invention provides an image classification method combining an FCN network and a Resnet network, further comprising: the middle layer of the FCN network module includes 5 convolutional pooling blocks and 2 convolutional activation blocks connected in sequence.

[0011] Furthermore, in order to obtain a more accurate classification model, the present invention provides an image classification method that combines an FCN network and a Resnet network, which also includes five convolutional pooling blocks. The first two convolutional pooling blocks respectively include two convolutional layers, an activation layer arranged after each convolutional layer, and a maximum pooling layer arranged after the last activation layer. The last three convolutional pooling blocks respectively include three convolutional layers, an activation layer arranged after each convolutional layer, and a maximum pooling layer arranged after the last activation layer.

[0012] Furthermore, in order to obtain a more accurate classification model, the present invention provides an image classification method that combines an FCN network and a Resnet network, further comprising: the number of output channels of the last convolutional activation block of the FCN network is equal to the number of input channels of the FCN network module.

[0013] Furthermore, in order to obtain a more accurate classification model, the present invention provides an image classification method that combines an FCN network and a Resnet network, and also includes the Resnet network module improved based on the Resnet-18 network model.

[0014] Furthermore, in order to obtain a more accurate classification model, the present invention provides an image classification method that combines an FCN network and a Resnet network, further comprising an intermediate layer of the Resnet network module including a convolutional layer, 4 convolutional residual blocks, a global average pooling layer, a fully connected layer and a classification layer connected in sequence, wherein each convolutional residual block includes multiple convolutional layers.

[0015] Furthermore, in order to perform classification more accurately, the present invention provides an image classification method that combines an FCN network and a Resnet network, and also includes preprocessing the image before inputting the image into the classification model, wherein the preprocessing is at least one processing method of cropping and scaling.

[0016] Furthermore, in order to obtain a more accurate classification model, the present invention provides an image classification method that combines an FCN network and a Resnet network, which also includes using a data augmentation method to amplify the images in the image data set when training the classification model. The data augmentation method is at least one of upper and lower mirroring, left and right mirroring, random rotation, and affine transformation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a FCN-32s network model in the prior art;

[0018] Figure 2 Schematic diagram of the Resnet-18 network model in the prior art;

[0019] Figure 3 Schematic diagram of the classification model of the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and technical effects of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] Example of an image classification method combining an FCN network and a Resnet network:

[0022] This embodiment provides an image classification method that combines an FCN network and a ResNet network. This image classification method that combines an FCN network and a ResNet network can be referred to as an image classification method. The image classification method according to this embodiment can solve the problem of insufficient classification accuracy of the ResNet classification network in the prior art.

[0023] The following is a detailed introduction to the classification model. Figure 1 A schematic diagram of a FCN-32s network model in the prior art; Figure 2 Schematic diagram of the Resnet-18 network model in the prior art; Figure 3 Schematic diagram of the classification model of the present invention.

[0024] In this embodiment, the classification model includes two parts, an FCN network module and a Resnet network module, which are connected in sequence. The input of the classification model is an image, and the output is a classification result.

[0025] 1. FCN network module. The FCN network module is used to add attention mechanism to each pixel of the input image of the input layer.

[0026] The FCN network module is obtained by improving the FCN network model. Specifically, the FCN network model includes an input layer, an intermediate layer, an upsampling layer, and an output layer. The output layer of the FCN network model is removed, and the input of the FCN network model (i.e., the input image) is multiplied by the output of the upsampling layer to obtain the FCN network module. The product obtained after the multiplication operation is the output of the FCN network module.

[0027] In the FCN network module, the FCN network model selects the FCN-32s network model, that is, the FCN network module is improved based on the FCN-32s network model. Figure 1 As shown in Figure 2, the input layer of the FCN-32s network model is used to receive images. The middle layer of the FCN-32s network model includes 5 convolutional pooling blocks and 2 convolutional activation blocks. Figure 1 As shown in the figure, among the five convolutional pooling blocks, the first two convolutional pooling blocks include two convolutional layers (Conv), an activation layer (ReLu layer) set after each convolutional layer, and a maximum pooling layer (max pooling) set after the last activation layer. The last three convolutional pooling blocks include three convolutional layers (Conv), an activation layer (ReLu layer) set after each convolutional layer, and a maximum pooling layer (max pooling) set after the last activation layer. Each convolutional pooling block is connected in sequence. Among the two convolutional activation blocks, each convolutional activation block includes a 1×1 convolutional layer and an activation layer (ReLu layer) set after the convolution layer. Each convolutional activation block is connected in sequence. Five convolutional pooling blocks, two convolutional activation blocks, and one upsampling layer are connected in sequence.

[0028] In the FCN network module, the upsampling layer of the FCN-32s network model uses a 32x upsampling process. 32x means the image is enlarged 32 times. Specifically, if the input layer of the FCN-32s network model receives a 32×32 image, after passing through five convolutional pooling blocks and two convolutional activation blocks, a 1×1 image is obtained. This is then upsampled through the upsampling layer of the FCN-32s network model to obtain a 32×32 image.

[0029] In addition, if the number of output channels of the upsampling layer of the FCN network model is not equal to the number of input channels (i.e., the number of channels of the input image), the number of output channels of the upsampling layer of the FCN network model is adjusted so that the number of output channels of the upsampling layer is equal to the number of input channels of the FCN network model. For example, if the image of the input layer is an RGB color image, the number of input channels is 3. If the number of output channels of the upsampling layer of the FCN network model is not 3, it is adjusted to 3. In this case, it is convenient to multiply the input of the FCN network model by the output of the upsampling layer, thereby obtaining a more accurate classification model.

[0030] In addition, if the number of output channels of the last convolutional activation block (Conv7+ReLu) in the middle layer is not equal to the number of input channels, the number of output channels of the last convolutional activation block of the FCN network model is adjusted so that the number of output channels of the last convolutional activation block is equal to the number of input channels of the FCN network model.

[0031] In the FCN network module, the output layer of the FCN-32s network model outputs the segmentation result. The segmentation result is presented as an image. By removing the output layer, the model input (i.e., the image fed into the FCN-32s network model) is multiplied by the output of the upsampling layer (i.e., the image obtained after upsampling) to obtain the product. This product becomes the output of the FCN network module. In this case, an attention mechanism can be added to each pixel of the input image, thereby improving the classification accuracy of the classification model.

[0032] 2. Resnet network module: The Resnet network module is used to classify images with the attention mechanism.

[0033] The Resnet network module is derived from an improved Resnet network model. Specifically, the Resnet network model consists of an input layer, intermediate layers, and an output layer. The input layer of the Resnet network model is removed to obtain the Resnet network module. The input of the Resnet network module is the product of the output of the FCN network module, that is, the input of the Resnet network module is the image with the attention mechanism added.

[0034] In the Resnet network module, the Resnet network model selects the Resnet-18 network model, that is, the Resnet network module is improved based on the Resnet-18 network model. Figure 2 As shown in the figure, the middle layer of the Resnet-18 network model includes a convolutional layer (conv1), 4 convolutional residual blocks, a global average pooling layer (average pool), a fully connected layer (fc) and a classification layer (softmax). Each convolutional residual block includes multiple convolutional layers. For example, the first convolutional residual block conv2-x connected to the convolutional layer conv1, where x represents that there are x convolutional layers in the convolutional residual block. Each convolutional residual block is connected in sequence. The convolutional layer, 4 convolutional residual blocks, the global average pooling layer, the fully connected layer and the classification layer are connected in sequence. The convolutional layer (conv1) is connected to the upsampling layer of the FCN network module, that is, the product result output by the FCN network module is input into the convolutional layer (conv1) of the Resnet-18 network model. In this way, a more accurate classification model can be obtained.

[0035] In the Resnet network module, such as Figure 2 As shown in Figure 2, the output layer of the Resnet-18 network model outputs the classification results.

[0036] After building the classification model introduced above, the classification model can be trained using the image dataset and the corresponding classification results to obtain a trained classification model.

[0037] The image dataset used for training consists of pre-collected images. Multiple images are collected from a variety of fields, such as natural scenes, medicine, or remote sensing. Each image in the image dataset has a corresponding classification result (i.e., category), which is annotated with a label.

[0038] In this embodiment, the image dataset is divided into a training set and a test set. Both the training set and the test set consist of multiple images. For example, the image dataset used for training is the Cifar10 dataset. The Cifar10 dataset has 10 categories of images, each with 6,000 images. 5,000 images are taken from each category, for a total of 50,000 images as the training set, and the remaining 10,000 images are used as the test set.

[0039] In this embodiment, all images in the training and test sets must be of the same size. If the images are not of the same size when they are collected, they are preprocessed to make them of the same size. This preprocessing can include at least one of cropping and scaling. This allows for more accurate classification.

[0040] In this embodiment, data augmentation is performed on the preprocessed training set. This augmentation method includes at least one of top-down mirroring, left-right mirroring, random rotation, and affine transformation. This helps enhance the generalization ability of the model training results, thereby improving the accuracy of the model's classification results. Furthermore, no data augmentation is performed on the test set.

[0041] In this embodiment, the entire training process includes initial training and parameter testing, wherein the training set is used for the initial training of the model, and the test set is used for the parameter testing of the model.

[0042] Before training, set the training hyperparameters for the classification model. For example, use the cross-entropy loss function for the classification model, optimize the loss function using stochastic gradient descent with momentum 0.9 and weight decay 0.0001. Initialize the learning rate to 0.1, and reduce the learning rate to one-tenth of its original value every 60 training epochs, for a total of 300 training epochs.

[0043] After setting the hyperparameters, enter the initial training process. During the initial training process, take out 50 labeled images from the training set each time, input Figure 3 For the classification model shown, when all images in the training set are trained once, one round of training is completed, and then the next round of training is repeated. After completing 300 rounds of training, the initial training ends.

[0044] After completing the initial training, enter the parameter test. During the parameter test process, take out, for example, 50 images each time from the test set and input them into the classification model after initial training to obtain the classification results. If the classification result of an image obtained by the model is consistent with the category marked by its label, the classification of the image is correct, otherwise it is wrong. After all the pictures in the test set are tested, the number of correctly classified images is calculated and counted as x. The total number of test sets is y, and the image classification accuracy p = x / y can be calculated. If the classification accuracy p is less than the preset threshold, continue to use the training set (some new images can be added) to train the classification model until the classification accuracy p is greater than or equal to the preset threshold, thereby obtaining a trained classification model.

[0045] In the image classification method of the combined FCN network and ResNet network of this embodiment, the trained classification model is used to classify the obtained image to be classified. The specific process of the classification method is as follows:

[0046] Step 1: Get the image to be classified.

[0047] In step 1, the images to be classified may belong to various fields such as natural scenes, medicine, or remote sensing. The field to which the images to be classified belong is the same as the field of the image dataset used to train the classification model.

[0048] In step 1, if the size of the image to be classified does not meet the size requirement of the input layer of the trained classification model, the image to be classified is preprocessed so that the size of the image to be classified meets the requirement. The preprocessing is at least one of cropping and scaling.

[0049] Step 2: Input the obtained image to be classified into the trained classification model for classification to obtain the classification result.

[0050] Based on the image classification method combining the FCN network and the Resnet network of this embodiment, a trained classification model is used to classify the image to be classified. In this method, the output of the upsampling layer of the FCN network model is multiplied by the input of the FCN network model using the FCN network model without the output layer and the Resnet network model without the input layer. The product result is output to the middle layer of the Resnet network model, thereby obtaining a classification model formed by combining the FCN network and the Resnet network. In this case, the product result of the FCN network (i.e., the image with the attention mechanism added) is used as the input of the middle layer of the Resnet network model without the input layer, thereby enhancing the spatial attention of the Resnet network and improving the classification accuracy of the classification model. Thus, the problem of insufficient classification accuracy of the Resnet classification network in the prior art can be solved.

[0051] In this embodiment, when training the classification model, the training set of the image data set is first preprocessed and then data augmentation is performed. In other embodiments, data augmentation can be performed first and then preprocessing.

[0052] In this embodiment, when training the classification model, data augmentation is performed only on the training set of the image dataset. In other embodiments, data augmentation can be performed on the image dataset.

Claims

1. An image classification method combining FCN network and Resnet network, characterized in that: include: 1) Obtain the image to be classified; 2) Inputting the obtained image to be classified into a trained classification model for classification to obtain a classification result, wherein the classification model is jointly formed by an FCN network module and a Resnet network module, wherein the FCN network module includes an input layer, an intermediate layer and an upsampling layer connected in sequence, and the Resnet network module includes an intermediate layer and an output layer connected in sequence, and the output of the upsampling layer of the FCN network module and the input of the FCN network module are multiplied to obtain a result as the input of the intermediate layer of the Resnet network module; using the image data set and the corresponding classification results to train the classification model to obtain a trained classification model.

2. The image classification method of combining FCN network and Resnet network according to claim 1 is characterized in that: The number of output channels of the upsampling layer is equal to the number of input channels of the FCN network module.

3. The image classification method of combining FCN network and Resnet network according to claim 1, characterized in that: The FCN network module is improved based on the FCN-32s network model.

4. The image classification method of combining FCN network and Resnet network according to claim 3 is characterized in that: The middle layer of the FCN network module includes 5 convolutional pooling blocks and 2 convolutional activation blocks connected in sequence.

5. The image classification method of combining FCN network and Resnet network according to claim 4 is characterized in that: Among the five convolutional pooling blocks, the first two convolutional pooling blocks include two convolutional layers, an activation layer after each convolutional layer, and a maximum pooling layer after the last activation layer, and the last three convolutional pooling blocks include three convolutional layers, an activation layer after each convolutional layer, and a maximum pooling layer after the last activation layer.

6. The image classification method of combining FCN network and Resnet network according to claim 4, characterized in that: The number of output channels of the last convolutional activation block of the FCN network is equal to the number of input channels of the FCN network module.

7. The image classification method of combining FCN network and Resnet network according to claim 1, characterized in that: The Resnet network module is improved based on the Resnet-18 network model.

8. The image classification method of combining FCN network and Resnet network according to claim 7, characterized in that: The middle layer of the Resnet network module includes a sequentially connected convolutional layer, 4 convolutional residual blocks, a global average pooling layer, a fully connected layer and a classification layer, wherein each convolutional residual block includes multiple convolutional layers.

9. The image classification method combining FCN network and Resnet network according to claim 1, characterized in that: Before inputting the image into the classification model, the image is preprocessed, and the preprocessing is at least one of cropping and scaling.

10. The image classification method of combining FCN network and Resnet network according to claim 9, characterized in that: When training the classification model, a data augmentation method is used to augment the images in the image data set, and the data augmentation method is at least one of upper and lower mirroring, left and right mirroring, random rotation, and affine transformation.

Images