Image classification performance improvement method based on multi-path output network architecture

Abstract

The present invention discloses a method for improving the performance of image classification based on a multi-path output network architecture, comprising the following steps: first, preprocessing the data, performing data enhancement, increasing the number and complexity of samples, and improving the robustness of the model. The preprocessed image is input into the multi-path output network architecture. In order to better achieve the performance of the network, two path selection algorithms are designed. During the training process, the two path selection algorithms are used together as classifiers of the network. Data enhancement is performed on the data, the data is input into the network, the performance of each path is compared, and the two path selection algorithms are combined to select the optimal path to obtain a predicted value. The advantages of the present invention are: solving the problem of network degradation caused by the increase in the number of layers, effectively avoiding the problems of gradient disappearance and gradient explosion, optimizing the performance of the network, and improving the recognition accuracy of image classification.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for improving image classification performance based on a multi-path output network architecture. Background Art

[0002] With the rapid development and popularization of Internet technology in the new era, people have greatly enriched their ways of obtaining information and rapidly reduced the cost of generating and obtaining information. At the same time, due to the widespread use of mobile devices such as mobile phones and tablets, people have generated a large amount of image data in their daily lives. Accordingly, the technology for image data processing has also attracted the attention of more and more researchers. In the research of traditional image classification methods (decision trees, random forests, support vector machines, etc.), [1] Most of the classifications are based on image features, using image processing algorithms to extract the corresponding features that have been expressed qualitatively or quantitatively, and then performing mathematical statistical analysis on these features or using classifiers to output classification results. However, it is difficult to process huge amounts of image data, and there are bottleneck problems such as extremely low efficiency, and it can no longer meet people's requirements for image classification accuracy and speed. [2] In recent years, with the development of artificial intelligence technology, deep learning technology has become more and more mature. Image classification based on deep convolutional neural network methods has broken through the bottleneck of traditional image classification methods and has become a research hotspot. [3] Compared with traditional image classification methods, it does not require manual feature description and extraction of the target image. Instead, it uses neural networks to autonomously learn features from training samples and extract higher-dimensional and abstract features. These features are closely related to the classifier, which solves the problem of manual feature extraction and classifier selection. [4] , and thus, the structure of deep convolutional neural networks for image classification is becoming more and more diverse, and its performance is far better than traditional image classification methods. However, most of the existing neural networks output results in the last layer, which wastes the information generated by the middle layer. The information is not fully utilized, and the deep neural network architecture is not flexible enough. Therefore, it is proposed to use a multi-path output method, set the multiple middle layer nodes we selected as the output layer, and propose a corresponding path selection algorithm to solve the problem of network degradation and improve the accuracy of the network. Network degradation: As the number of layers of deep convolutional neural networks increases, the network performance decreases instead of increases.

[0003] With the advent of the intelligent information age, deep learning came into being. As a branch of machine learning, deep learning aims to simulate the human neural network system to build a deep artificial neural network, analyze and interpret the input data, and combine the underlying features of the data into abstract high-level features. As a typical representative of deep learning, deep convolutional neural networks shine in computer vision tasks. Compared with traditional image classification algorithms that extract features manually, convolutional neural networks use convolution operations to extract features from input images, effectively learn feature expressions from a large number of samples, and have strong model generalization capabilities. For example, VGG, jointly developed by the Oxford Visual Geometry Group, a research team at the University of Oxford, and DeepMind, [5] Convolutional neural network, VGG network is a classic image classification network. The network structure is shown in the figure below. The VGG network model proposes to use a small 3*3 convolution kernel to replace the large 5*5 and 7*7 convolution kernels, which reduces the number of parameters and reduces the computational complexity. Liu Xuan from Fudan University [6] In 2018, a deep neural network-based transfer learning algorithm was proposed to solve the problem of lack of labeled high-resolution remote sensing images. The algorithm mainly uses VGGNet as the base classifier, uses labeled remote sensing data samples, and fine-tunes the top layer of the 16-layer VGG network through the back propagation algorithm to achieve good classification of remote sensing images.

[0004] The VGG network has achieved good results in both image classification and object positioning tasks, and the generalization of the VGG network is very good, but the increase in network depth will bring about the problem of network degradation with increased training error. Therefore, the optimal depth of the VGG network is limited to 16 to 19 layers.

[0005] He et al. [7] In 2019, Arief Budhiman proposed a residual convolutional neural network (ResNet), which deepened the network and improved the performance of image classification tasks. [8] ResNet was used to classify melanoma, and the validation results showed that the accuracy of the model reached 83%. [9] Importing pre-trained weights into the ResNet model, training the last three layers of the model, achieved good results in object detection and recognition. Gao et al.

[10] The proposed Res2Net network improves the multi-scale representation capability at a finer level, and Res2Net shows better performance in image classification tasks on multiple datasets.

[0006] The ResNet network deepens the number of network layers, effectively alleviating the network degradation problem and improving the accuracy of image classification. However, the results can only be output in the last layer, and the information carried by the intermediate layers during the network training process is not fully utilized, which correspondingly weakens the performance of the network.

[0007] In general, the prior art has the following defects:

[0008] (1) As the neural network continues to deepen, the network will have more powerful expression capabilities and layer-by-layer feature learning, and the performance will continue to improve. However, if it exceeds a certain threshold, the network will degenerate.

[0009] (2) Deep neural network training and testing do not fully utilize the information carried by the intermediate layers, which weakens the performance of the network.

[0010] (3) The network’s accuracy in image classification is not high.

[0011] References

[0012] [1] Zeng Fuliang, Hu Wenjin, He Guoyuan, Xue Panpan. Thangka image classification based on DenseNet[J]. Modern Electronic Technology, 2022, 45(06): 153-157. DOI: 10.16652 / j.issn.1004-373x.2022.06.028;

[0013] [2] Huang Qianfeng, Dong Qin. Research on crop disease image classification based on convolutional neural network[J]. Information and Computer (Theoretical Edition), 2022, (05): 138-142+146;

[0014] [3] Jiang Xiansen. A review of deep active learning research in image classification[J]. Yangtze River Information and Communication, 2021, 34(03): 161-163;

[0015] [4] Ji Changqing, Gao Zhiyong, Qin Jing, Wang Zumin. A review of image classification algorithms based on convolutional neural networks[J]. Journal of Computer Applications, 2022, 42(04): 1044-1049;

[0016] [5] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition [C] / / International Conference of LearningRepresentation, May 7-9, 2015, San Diego, CA.arXiv:1409.1556v6[cs.CV], 2015;

[0017] [6]X.Liu,M.Chi,Y.Zhang and Y.Qin,"Classifying High Resolution RemoteSensing Images by Fine-Tuned VGG Deep Networks,"IGARSS 2018-2018 IEEEInternational Geoscience and Remote Sensing Symposium,Valencia,Spain,2018,pp.7137-7140,doi:10.1109 / IGARSS.2018.8518078;

[0018] [7]He K M,Zhang X Y,Ren S Q and Sun J.2016a.Deep residual learningfor image recognition / / Proceedings of 2016IEEE Conference on Computer Visionand Pattern Recognition(CVPR).Las Vegas,USA:IEEE:770-778[DOI:10.1109 / CVPR.2016.90];

[0019] [8]Budhiman,S.Suyanto and A.Arifianto,"Melanoma Cancer ClassificationUsing ResNet with Data Augmentation,"2019International Seminar on Research ofInformation Technology and Intelligent Systems(ISRITI),Yogyakarta,Indonesia,2019,pp.17-20,doi:10.1109 / ISRITI48646.2019.9034624;

[0020] [9]Z.Zahisham, CPLee and KMLim, "Food Recognition with ResNet-50," 2020IEEE 2nd International Conference on Artificial Intelligence in Engineering and Technology (IICAIET), Kota Kinabalu, Malaysia, 2020, pp.1-5, doi:10.1109 / IICAIET49801.2020.9257825;

[0021]

[10] Gao SH, Chen MM, Zhao K, Zhang XY, Yang MH and TorrP. 2021. Res2Net: a new multi-scale backbone architecture. IEEETransactions onPattern Analysis and Machine Intelligence, 43(2):652-662[DOI:10.1109 / TPAMI.2019.2938758]. Summary of the invention

[0022] In view of the defects of the prior art, the present invention provides a method for improving image classification performance based on a multi-path output network architecture.

[0023] In order to achieve the above invention object, the technical solution adopted by the present invention is as follows:

[0024] A method for improving image classification performance based on a multi-path output network architecture comprises the following steps:

[0025] Step 1: First, preprocess the data and perform data enhancement to increase the number and complexity of samples and improve the robustness of the model.

[0026] Step 2: Input the preprocessed image into the multi-path output network architecture.

[0027] Step 3: In order to achieve better network performance, two path selection algorithms are designed. These two path selection algorithms are used together as classifiers of the network during the training process.

[0028] The first algorithm is: during network training, multiple image data are packaged into a batch, each batch is uniformly input into the network, and the network is output through different network layers. All the prediction values ​​obtained are directly cross-entropy with the labels, and then the output path with the smallest loss value among all the prediction values ​​is taken as the selection for this batch.

[0029] The second algorithm is to package multiple image data into a batch. For each data in a batch, N predicted tensor data are obtained through N output nodes. The largest data among the N predicted tensor data is taken as the predicted value of image classification, and this path is used as the path selected by this data, which is equivalent to each data in a batch selecting a path. These two path selection algorithms are used together as the classifier of the network during the training process.

[0030] Step 4: Output the path, perform data enhancement on the data, input the data into the network, compare the performance of each path, combine the two path selection algorithms, select the optimal path, and obtain the predicted value.

[0031] Furthermore, the data preprocessing described in step 1 includes:

[0032] ① For the original-size image, first fill it with 0s with a width of 4 units on all sides to increase the image size, and then randomly crop the expanded image to the target size.

[0033] ② The image entering the network training has a probability of flipping half and a probability of not flipping half.

[0034] ③The mean and variance of the image data are normalized using the standards of (0.4914, 0.4822, 0.4465) and (0.2023, 0.1994, 0.2010).

[0035] Furthermore, the multi-path output network architecture described in step 2 is a 50-layer neural network, in which 16 path output channels are set. The data input network first passes through a 7*7 convolution kernel with an output channel number of 64 and a step size of 2. After passing through a pooling layer with a step size of 2, the image size is further reduced; the following is a network block with three layers as a part, first using a 1*1 convolution kernel with an output channel number of 64, then using a 3*3 convolution kernel with an output channel number of 64, and finally using a 1*1 convolution to increase the dimension, and output once at this layer. There are 3 network blocks with this setting; 4 1*1 convolution kernels are set below. The number of output channels of the convolution kernel is 128, the number of output channels of the 3*3 convolution kernel is 128, and the number of output channels of the 1*1 convolution kernel is 512; then 6 network blocks with the number of output channels of the 1*1 convolution kernel is 256, the number of output channels of the 3*3 convolution kernel is 256, and the number of output channels of the 1*1 convolution kernel is 1024 are set; finally, 3 network blocks with the number of output channels of the 1*1 convolution kernel is 512, the number of output channels of the 3*3 convolution kernel is 512, and the number of output channels of the 1*1 convolution kernel is 2048 are set, and the network sets the output in the last layer of each network block.

[0036] Furthermore, in order to make better use of the information in the middle layer, the data is input from the first layer and output in the middle layer selected by the network instead of being fixed at the last layer. All prediction results obtained by aggregating data through different paths are finally selected as the prediction value of this round of data training.

[0037] Preferably, in step 4, when the number of network layers is 18, 18 output paths are set.

[0038] Preferably, in step 4, when the number of network layers is 34, 34 output paths are set.

[0039] Preferably, in step 4, when the number of network layers is 50, 16 output paths are set.

[0040] Compared with the prior art, the advantages of the present invention are:

[0041] The multi-path output network architecture has a very good containment effect on the network degradation problem caused by the increase in the number of layers of deep convolutional networks, and effectively solves such problems. At the same time, under this network architecture, the problems of gradient disappearance and gradient explosion can also be effectively avoided.

[0042] On the other hand, by optimizing the performance of the network, when using a network architecture with multi-path output, the utilization of intermediate layer information can be effectively enhanced. Compared with the ordinary neural network with direct connection, the designed network architecture is added, which can improve the recognition accuracy of image classification by 6 to 7 percentage points. At the same time, for other networks, such as Res Net, there is also a significant improvement in image classification performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the structure of a multi-path output network architecture according to an embodiment of the present invention;

[0044] Figure 2 is a flow chart of a method for improving image classification performance according to an embodiment of the present invention;

[0045] Figure 3 It is a schematic diagram of the experimental process of the method for improving image classification performance according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0047] First, the data is preprocessed and data enhancement is performed to increase the number and complexity of samples and improve the robustness of the model.

[0048] ① For the original-size image, first fill it with 0s with a width of 4 units on all sides to increase the image size, and then randomly crop the expanded image to the target size.

[0049] ② The image entering the network training has a probability of flipping half and a probability of not flipping half.

[0050] ③The mean and variance of the image data are normalized using the standards of (0.4914, 0.4822, 0.4465) and (0.2023, 0.1994, 0.2010).

[0051] Then, the preprocessed image is input into the multi-path output network architecture. In order to make better use of the information in the middle layer, the data can be input from the first layer, but it does not have to be fixed at the last layer output. Instead, it can be output at the middle layer selected by the network, aggregating all the prediction results obtained by the data through different paths, and finally selecting the prediction result of the path with the minimum cost as the prediction value of this round of data training. Figure 1As shown, this embodiment constructs a 50-layer neural network, in which 16 path output channels are set. The data input network first passes through a 7*7 convolution kernel, the number of output channels is 64, and the step size is 2. After passing through a pooling layer with a step size of 2, the image size is further reduced; the following is a network block with three layers as a part, first using a 1*1 convolution kernel with an output channel number of 64, then using a 3*3 convolution kernel with an output channel number of 64, and finally using a 1*1 convolution to increase the dimension, and output once in this layer. There are 3 network blocks with this setting; 4 1*1 convolution kernels with an output channel number of 128 are set below, The number of output channels of the convolution kernel of 3*3 size is 128, and the number of output channels of the convolution kernel of 1*1 size is 512; then 6 network blocks with the number of output channels of the convolution kernel of 1*1 size is 256, the number of output channels of the convolution kernel of 3*3 size is 256, and the number of output channels of the convolution kernel of 1*1 size is 1024 are set; finally, 3 network blocks with the number of output channels of the convolution kernel of 1*1 size is 512, the number of output channels of the convolution kernel of 3*3 size is 512, and the number of output channels of the convolution kernel of 1*1 size is 2048 are set, and the network sets the output layer, that is, the average pooling layer and the linear layer, at the last layer of each network block. The data output network mode under the network architecture of multi-path output of the present invention has a higher degree of freedom.

[0052] Secondly, in order to better achieve the performance of the network, we designed two path selection algorithms. Taking the 50-layer multi-path output neural network as an example, the first algorithm is to input each batch (in network training, multiple image data are usually packaged into a batch and uniformly input into the network) into the network, output the network through different network layers, and directly perform cross entropy on all the predicted values ​​and labels, and then take the output path with the smallest loss value among all the predicted values ​​as the selection for this batch. The second algorithm is that for each data in a batch, 16 predicted tensor data can be obtained through 16 output nodes, and the largest data among these 16 predicted data is taken as the predicted value for image classification, and this path is taken as the path selected by this data, which is equivalent to each data in a batch selecting a path. In the training process, these two path selection algorithms are used together as the classifier of the network. In the test phase, since there is no labeled data, only the second path selection algorithm is used as the basis for network image classification.

[0053] As mentioned above, the image classification method is used to verify the performance of the proposed network architecture with different numbers of layers, for example:

[0054] When the number of network layers is 18, 18 output paths are set. First, data enhancement is performed on the data, and then the data is input into the network. The performance of each path is compared, and the two path selection algorithms are combined to select the optimal path and obtain the prediction value;

[0055] When the number of network layers is 34, 34 output paths are set. First, data enhancement is performed on the data, and then the data is input into the network. The performance of each path is compared, and the two path selection algorithms are combined to select the optimal path and obtain the prediction value;

[0056] When the number of network layers is 50, in order to reduce the amount of calculation and improve the convergence speed of training, 16 output paths are set. First, the data is enhanced, and then the data is input into the network. The performance of each path is compared, and the two path selection algorithms are combined to select the optimal path to obtain the prediction value.

[0057] like Figure 2 and 3 As shown in the figure, the deep convolutional neural network model is trained and tested on the dataset. The quality of the model is measured based on the classification results obtained on the dataset, so it is crucial to use a good dataset. CIFAR-10 is selected as the dataset. The CIFAR-10 dataset contains 60,000 color images with a resolution of 32×32 pixels. It consists of 10 categories: airplane, horse, dog, bird, cat, deer, frog, horse, ship, and truck. The 10 categories are independent of each other and there is no overlap. The training set contains 50,000 samples and 50,000 labels, and the test set contains 10,000 samples and 10,000 labels.

[0058] For the 32*32 size images in CIFAR-10, we first fill the four edges with 0s and then randomly crop the expanded images to 32*32. Then we flip the 32*32 images with a probability of 50% and not flip them with a probability of 50%. Finally, we normalize the mean and variance of the image data with the standard of (0.4914, 0.4822, 0.4465) and (0.2023, 0.1994, 0.2010).

[0059] The experiment constructed 18-layer, 34-layer, and 50-layer multi-path output networks and 18-layer, 34-layer, and 50-layer ResNet networks. The 18-layer multi-path output network and the 34-layer multi-path input network had 18 and 34 path choices respectively. In order to reduce the amount of calculation and improve the convergence speed of training, the 50-layer network was set to only 16 path choices. In the data-enhanced CIFAR-10 dataset, the multi-path output neural network has an average Top-1 accuracy improvement of 3 to 4 percentage points over the ResNet network. At the same time, the multi-output neural network did not have the network degradation problem caused by the increase in the number of layers at 18, 34, and 50 layers. In the above experiment, the multi-path output neural network has better image classification performance than ResNet in the horizontal comparison of the same number of layers, and shows an excellent containment effect on the network degradation problem in the vertical comparison.

[0060] On the data-enhanced CIFAR-10 dataset, taking a 50-layer multi-path output neural network as an example, compared with a 50-layer ordinary neural network with the same settings but without multi-path output, the multi-path output neural network can improve the recognition accuracy of image classification by 6 to 7 percentage points.

[0061] On the data-enhanced CIFAR-10 dataset, taking the 50-layer multi-path output neural network as an example, compared with the 50-layer ResNet network, the multi-path output neural network can improve the recognition accuracy of image classification by an average of 3 to 4 percentage points.

[0062] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the implementation methods of the present invention, and should be understood that the protection scope of the present invention is not limited to such special statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A method for improving image classification performance based on a multi-path output network architecture, characterized in that: The following steps are involved: Step 1: First, preprocess the data and perform data enhancement to increase the number and complexity of samples and improve the robustness of the model; Step 2, inputting the preprocessed image into the multi-path output network architecture; The multi-path output network architecture is a 50-layer neural network, in which 16 path output channels are set. The data input network first passes through a 7*7 convolution kernel with an output channel number of 64 and a step size of 2. After passing through a pooling layer with a step size of 2, the image size is further reduced; the following is a network block with three layers as part, first using a 1*1 convolution kernel with an output channel number of 64, then using a 3*3 convolution kernel with an output channel number of 64, and finally using a 1*1 convolution to increase the dimension, and output once in this layer. There are 3 network blocks with this setting; the following sets of 4 1*1 convolution kernels with an output channel number of 128 and 3 *3 size convolution kernel output channel number is 128, 1*1 size convolution kernel output channel number is 512 network blocks; then set up 6 1*1 size convolution kernel output channel number is 256, 3*3 size convolution kernel output channel number is 256, 1*1 size convolution kernel output channel number is 1024 network blocks; finally set up 3 1*1 size convolution kernel output channel number is 512, 3*3 size convolution kernel output channel number is 512, 1*1 size convolution kernel output channel number is 2048 network blocks, and the network sets the output layer at the last layer of each network block, that is, an average pooling layer and a linear layer; Step 3: In order to achieve better network performance, two path selection algorithms are designed; during the training process, these two path selection algorithms are used together as classifiers of the network; The first algorithm is: during network training, multiple image data are packaged into a batch, each batch is uniformly input into the network, and the network is output through different network layers. All the predicted values ​​and labels are directly cross-entropy, and then the output path with the smallest loss value among all the predicted values ​​is selected as the selection for this batch; The second algorithm is: the network packages multiple image data into a batch. For each data in a batch, N predicted tensor data are obtained through N output nodes. The largest data among the N predicted tensor data is taken as the predicted value of image classification, and this path is used as the path selected by this data, which is equivalent to each data in a batch selecting a path. In the training process, these two path selection algorithms are used together as the classifier of the network. In the test phase, since there is no labeled data, only the second path selection algorithm is used as the basis for network image classification. Step 4: Output the path, perform data enhancement on the data, input the data into the network, compare the performance of each path, combine the two path selection algorithms, select the optimal path, and obtain the predicted value.

2. The method for improving image classification performance based on a multi-path output network architecture according to claim 1, characterized in that: The data preprocessing described in step 1 includes: ① For the original-size image, fill it with 0s with a width of 4 units on all sides to increase the image size, and then randomly crop the expanded image to the target size; ② The image entering the network training has a probability of flipping half and a probability of not flipping half; ③The mean and variance of the image data are normalized using the standards of (0.4914, 0.4822, 0.4465) and (0.2023, 0.1994, 0.2010).

3. The method for improving image classification performance based on a multi-path output network architecture according to claim 1, characterized in that: In order to make better use of the information in the middle layer, the data is input from the first layer and output in the middle layer selected by the network instead of being fixed at the last layer. All prediction results obtained by aggregating data through different paths are finally selected as the prediction value of this round of data training.

4. The method for improving image classification performance based on a multi-path output network architecture according to claim 1, characterized in that: In step 4, when the number of network layers is 18, 18 output paths are set.

5. The method for improving image classification performance based on a multi-path output network architecture according to claim 1, characterized in that: In step 4, when the number of network layers is 34, 34 output paths are set.

6. The method for improving image classification performance based on a multi-path output network architecture according to claim 1, characterized in that: In step 4, when the number of network layers is 50, 16 output paths are set.

Images