Fruit image classification method based on deep transfer learning
Through the deep transfer learning method, the pre-trained model and model fusion technology are used to solve the problems of huge parameters, slow training speed and poor generalization in fruit image classification, and the fruit with high accuracy and stability are achieved Image classification.
Patent Information
- Application Number
- CN202210524205.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-13
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-05-13
AI Technical Summary
Traditional deep learning methods have problems such as redundant, huge parameters, slow convergence speed of training process, low robustness and poor generalization in fruit image classification, and require a large number of data sets and high computing power.
Using the deep transfer learning method, the pre-trained VGG16, InceptionV3 and ResNet50 models are used to freeze the convolutional layer parameters and modify the fully connected layer structure to construct a new migration model TL-VGG16, TL-InceptionV3 and TL-ResNet50, and weighted The average method and the relative majority voting method are combined.
It improves the recognition accuracy and stability of fruit image classification, alleviates the problems of gradient vanishing and gradient explosion, enhances the robustness and generalization of the model, and reduces training time and calculation needs.
Smart Images

Figure CN114881155B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of agricultural informatization and relates to a fruit image classification method based on deep transfer learning. Background Art
[0002] In terms of image classification technology required in sorting systems such as fruit classification, maturity judgment and quality selection, the most representative one is the deep learning method that directly learns complex feature expressions from massive data. The classification process is generally achieved by collecting a large amount of raw data, annotating the raw data, and then establishing a specific deep convolutional neural network model for training for different categories of images.
[0003] However, it is very difficult to use deep learning to collect images from specific fields and train classifiers from scratch. For specific categories, specific models are required, which are not very adaptable, and the early data collection is time-consuming and labor-intensive. Deep networks also have problems such as long training time, large amount of calculation, and high equipment requirements. Ordinary research equipment cannot reach its computing power, which to a certain extent limits the generalization of deep image recognition in various fields.
[0004] Therefore, at present, we need to focus on using the trained universal model to meet the personalized needs of the specific model, and deep transfer learning can complete this process. Transfer learning can be simply described as the successful application of image classification knowledge learned on a large dataset to the new target classification task to be solved. On the one hand, building a specific model based on the existing best network by fine-tuning the network layer structure is faster and easier than building and training a new randomly initialized network model; on the other hand, the pre-trained model has already learned a wealth of features and does not require a large amount of data to be retrained, which can effectively shorten the training time. Summary of the invention
[0005] In view of this, the purpose of the present invention is to provide a fruit image classification method based on deep transfer learning, which solves the problems of traditional target classification models being complicated, large parameters, slow convergence speed of the training process, large training errors caused by the influence of the environment on the image, low robustness, and poor generalization. It also overcomes the shortcomings of general network models based on deep neural network training, such as the need to collect a large amount of data sets, long calculation time, and high requirements on computer performance.
[0006] In order to achieve the above object, the present invention provides the following technical solutions:
[0007] The fruit image classification method based on deep transfer learning specifically includes the following steps:
[0008] S1: Download the initial weights of VGG16, InceptionV3, and ResNet50 models pre-trained on the ImageNet dataset.
[0009] S2: The first 13 convolutional layers of the VGG16 network are built according to the original model, and all the parameters of the convolutional layers are frozen so that the parameters of the convolutional layers do not participate in the training. Then the top three fully connected layers are modified, that is, the number of parameters of the first fully connected layer is halved, the second fully connected layer is removed, and the output dimension of the third fully connected layer is changed from 1000 to the number of target categories. A new migration model TL-VGG16 network is proposed;
[0010] S3: Build the convolution layer of the InceptionV3 model according to the original model, freeze all the parameters of the convolution layer, so that the convolution layer does not participate in the training, and remove the auxiliary classifier 1 (aux_1) after Inception_4a and the auxiliary classifier 2 (aux_2) after Inception_4d. Then reduce the dimension of the final output layer, i.e. the main classifier, to the actual number of classifications, and propose a new migration model TL-InceptionV3.
[0011] S4: Build the convolution layer of the ResNet50 model according to the original model, and freeze all the parameters of the convolution layer so that the convolution layer does not participate in the training. Then modify the average pooling downsampling layer and the fully connected layer. That is, replace the 7×7 global average pooling layer after the convolution layer with two 3×3 convolution structures with a stride of 2. They have the same receptive field. Use the strong detail extraction ability of the convolution layer to deepen the detail features. Then add a fully connected layer to reduce the output layer dimension to the number of data set categories. A new migration model TL-ResNet50 is proposed.
[0012] S5: Process the images collected under natural conditions according to the image processing method of the ImageNet public dataset, and then input the processed images into the improved migration models in steps S2, S3 and S4 for training respectively;
[0013] S6: The migration model trained in step S5 is fused by weighted average method, and the category probabilities output by each model are added and averaged;
[0014] S7: Based on the weighted average model fusion framework in step S6, the prediction image is input to perform predictions on the three migration models, and the prediction probabilities are output. The three prediction probabilities are weighted averaged to output the fused prediction probabilities.
[0015] S8: The models trained in step S5 are integrated by relative majority voting method, following the principle of minority obeys majority, and the prediction category with the most votes is taken as the output category;
[0016] S9: Based on the relative majority voting model fusion framework in step S8, the prediction image is input to perform predictions on the three migration models, the prediction probabilities are output, and the prediction result with the most votes is selected from the three prediction probabilities, and the fused probability is output.
[0017] Furthermore, in step S5, the input image needs to be batch normalized (BN) to accelerate the convergence of the network so that the feature map data satisfies the distribution law with a mean of 0 and a variance of 1. The calculation formula is as follows:
[0018]
[0019] Among them, E[x (k) ] refers to the average value of each neuron input value when training a batch of data; Refers to the standard deviation of the input values of each neuron when training a batch of data. The input x is a color image with three channels of RGB, channels is 3, x = (x (1) ,x (2) ,x (3) ), x (1) 、x (2) 、x (3) Represent the feature matrices corresponding to the R, G, and B channels respectively.
[0020] Furthermore, in step S5, the input image needs to be preprocessed. When using the existing pre-trained model parameters in transfer learning, the same image preprocessing method must be used as the original model. The original model was trained on ImageNet. During training, the mean values of the three channels R, G, and B were subtracted respectively. The calculation formula is:
[0021] mean=(0.485,0.456,0.406)×255
[0022] R_mean = 123.68
[0023] G_mean = 116.78
[0024] B_mean = 103.94
[0025] img=img-[R_mean,G_mean,B_mean]
[0026] Among them, (0.485, 0.456, 0.406) is the mean of the ImageNet dataset.
[0027] Furthermore, in step S6, the core idea of the weighted average model fusion is to add and average the category probabilities output by each model for multiple different models. The calculation formula is:
[0028]
[0029] Furthermore, in step S8, the relative majority voting model fusion strategy usually takes the category with the most votes as the prediction result and becomes the final classification category. If more than one category obtains the highest votes, one of them is randomly selected as the final classification category. The calculation formula is:
[0030]
[0031] The beneficial effects of the present invention are:
[0032] 1) The present invention can solve the recognition problems of irregular fruit shapes, large changes in natural lighting conditions, and obvious differences in growth conditions in natural environments, and can effectively alleviate the problems of gradient vanishing and gradient exploding, and has good robustness and generalization.
[0033] 2) The present invention adopts three classic networks, VGG16, InceptionV3 and ResNet50, as the backbone networks, and uses the transfer learning method to migrate the low-level parameters of these three networks to achieve its own classification purpose, with high recognition accuracy and stability.
[0034] 3) The present invention modifies and optimizes the high-level structures and parameters of VGG16, InceptionV3 and ResNet50 networks, and proposes new migration models TL-VGG16, TL-InceptionV3 and TL-ResNet50, which can solve the overfitting problem of traditional CNN networks in the training process of small data sets, reduce the number of parameters and improve the training speed.
[0035] 4) The ResNet50 network in the ResNet series of networks with the best classification effect in the CV field uses two small convolution kernels with a stride of 2 to replace the global average pooling layer of the ResNet50 model. The two have the same receptive field, but replacing the pooling layer with a convolution layer can deepen the extraction of details and make up for the disadvantage of the low-level convolution layer that the extracted features become less due to the unchanged parameters.
[0036] 5) The present invention uses two model fusion methods, weighted average method and relative majority voting method, to fuse the three migration models, integrate the recognition capabilities of the models, improve the fixed mode of single model input and single model output, and solve the problem that different models have different advantages and disadvantages and different applicability.
[0037] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:
[0039] Figure 1 It is a schematic diagram of the overall process of the fruit image classification method based on deep transfer learning of the present invention;
[0040] Figure 2 It is a network structure diagram of the TL-VGG16 model of the fruit image classification method based on deep transfer learning of the present invention;
[0041] Figure 3 It is a network structure diagram of the TL-InceptionV3 model of the fruit image classification method based on deep transfer learning of the present invention;
[0042] Figure 4 It is a network structure diagram of the TL-ResNet50 model of the fruit image classification method based on deep transfer learning of the present invention;
[0043] Figure 5 The present invention is a flow chart of the model fusion based on the weighted average method of three migration models;
[0044] Figure 6 This is a flow chart of the fusion of the relative majority voting model based on three migration models in the present invention. DETAILED DESCRIPTION
[0045] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.
[0046] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.
[0047] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.
[0048] See also Figure 1 to Figure 6 The present invention preferably provides a flow chart of a fruit image classification method based on deep transfer learning, which includes the following specific steps:
[0049] (1) Collect images of various types of fruits in natural environments, process the images according to the image processing method of the ImageNet public dataset, divide them into training sets and test sets with a certain proportion, and classify all images by labels;
[0050] (2) Download the initial weights of the VGG16, InceptionV3, and ResNet50 models pre-trained on the ImageNet dataset.
[0051] (3) Build the TL-VGG16 network model structure in the Pycharm tool under the TensorFlow2.1 environment. First, freeze the parameters of the first 13 convolutional layers so that the convolutional layers do not participate in training, then halve the parameters of the first fully connected layer, remove the second fully connected layer, and change the output dimension of the third fully connected layer from 1000 to the actual number of categories;
[0052] (4) Build the TL-InceptionV3 network model structure in the Pycharm tool under the TensorFlow2.1 environment. First, freeze the parameters of the regular convolutional layer and the convolutional layer of the Inception module. Since the low-level parameters are frozen, there is no need for auxiliary classifiers to adjust the low-level parameters. Therefore, remove the two auxiliary classifiers in the Inception module and only use the last main classifier. Change the output dimension of the main classifier from 1000 to the actual number of classifications.
[0053] (5) Build the TL-ResNet50 network model structure in the Pycharm tool under the TensorFlow2.1 environment. First, freeze the convolution layer parameters in BasicBlock and Bottlneck, and then replace the 7×7 global average pooling layer connected to the convolution layer with two 3×3 small convolutions with a stride of 2 to make up for some of the detailed features lost due to the freezing of the parameters of the lower convolution layer and increase the feature extraction capability. Then add a fully connected layer and activate it with the ReLU activation function. Finally, change the output dimension of the output layer from 1000 to the actual number of categories.
[0054] (6) For the migration models defined in steps (3), (4), and (5), input the training set and the test set for training, visualize each migration model through the TensorBoard toolkit, and output the accuracy and loss curve graphs of each migration model;
[0055] (7) The trained TL-VGG16, TL-InceptionV3 and TL-ResNet50 migration models are fused by weighted averaging, and then the category probabilities output by the three models are added and averaged to output the final prediction result;
[0056] (8) The trained TL-VGG16, TL-InceptionV3 and TL-ResNet50 migration models are fused through the relative majority voting method. After each classifier outputs its predicted category, the one with the most votes is usually used as the prediction result of the sample and becomes the final classification category. If more than one category receives the highest votes, one of them is randomly selected as the final classification category.
[0057] (9) Based on the prediction results of the two fusion models, compare the advantages and disadvantages of the two strategies, and analyze the practicality and adaptability of the two fusion strategies.
[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.
Claims
1. A fruit image classification method based on deep transfer learning, characterized by: The method specifically comprises the following steps: S1: Download the initial weights of VGG16, InceptionV3 and ResNet50 models pre-trained on the ImageNet dataset; S2: The first 13 convolutional layers of the VGG16 network are built according to the original model, the initial weights are referenced and all parameters of the convolutional layers are frozen so that the parameters of the convolutional layers do not participate in the training. Then the top three fully connected layers are modified, that is, the number of parameters of the first fully connected layer is halved, the second fully connected layer is removed, and the output dimension of the third fully connected layer is changed from 1000 to the number of target categories. A new migration model TL-VGG16 network is proposed; S3: Build the convolution layer of the InceptionV3 model according to the original model, reference the initial weights and freeze all the parameters of the convolution layer so that the convolution layer does not participate in the training, and remove the auxiliary classifier 1 after Inception_4a, i.e. aux_1, and the auxiliary classifier 2 after Inception_4d, i.e. aux_2. Then reduce the dimension of the final output layer, i.e. the main classifier, to the actual number of classifications, and propose a new migration model TL-InceptionV3. S4: Build the convolution layer of the ResNet50 model according to the original model, reference the initial weights and freeze all parameters of the convolution layer so that the convolution layer does not participate in training, then modify the average pooling downsampling layer and the fully connected layer, that is, replace the 7×7 global average pooling layer after the convolution layer with two 3×3 convolution structures with a stride of 2, add a fully connected layer, reduce the dimension of the output layer to the number of data set categories, and propose a new migration model TL-ResNet50; S5: Process the images collected under natural conditions according to the image processing method of the ImageNet public dataset, and then input the processed images into the improved migration models in steps S2, S3 and S4 for training respectively; S6: The models trained in step S5 are fused by weighted averaging, and the class probabilities output by each model are added and averaged; S7: Based on the weighted average model fusion framework in step S6, the prediction image is input to perform predictions on the three migration models, and the prediction probabilities are output. The three prediction probabilities are weighted averaged to output the fused prediction probabilities. S8: The models trained in step S5 are integrated by relative majority voting method, following the principle of minority obeys majority, and the prediction category with the most votes is taken as the output category; S9: Based on the relative majority voting model fusion framework in step S8, the prediction image is input to perform predictions on the three migration models, the prediction probabilities are output, and the prediction result is selected with the most votes among the three prediction probabilities, and the fused prediction probabilities are output.
2. The fruit image classification method based on deep transfer learning according to claim 1, characterized in that: In S1, the classical network is used as a springboard to migrate several weights trained on ImageNet to classification tasks for different fruits, and unified training is performed on images of different types of fruits in different environments to identify the type, color, shape and freshness of the fruits.
3. The fruit image classification method based on deep transfer learning according to claim 2, characterized in that: In S5, the input image needs to be batch normalized (BN) to accelerate the convergence of the network so that the data of the feature map satisfies the distribution law with a mean of 0 and a variance of 1. The calculation formula is as follows: Among them, E[x (k) ] refers to the average value of each neuron input value when training a batch of data; Refers to the standard deviation of the input values of each neuron when training a batch of data; the input x is a color image with three channels of RGB, channels is 3, x=(x (1) ,x (2) ,x (3) ), x (1) 、x (2) 、x (3) Represent the feature matrices corresponding to the R, G, and B channels respectively.
4. The fruit image classification method based on deep transfer learning according to claim 3, characterized in that: In S5, the input image is preprocessed, and the existing pre-trained model parameters are used in transfer learning. The same image preprocessing method is used as the original model. The original model is trained on ImageNet. During training, the mean values of the three channels R, G, and B are subtracted respectively. The calculation formula is: mean=(0.485,0.456,0.406)×255 R_mean = 123.68 G_mean = 116.78 B_mean = 103.94 img=img-[R_mean,G_mean,B_mean] Among them, (0.485, 0.456, 0.406) is the mean of the ImageNet dataset.
5. The fruit image classification method based on deep transfer learning according to claim 4, characterized in that: In S6, the core idea of the weighted average model fusion is to add and average the category probabilities output by each model for multiple different models. The calculation formula is:
6. The fruit image classification method based on deep transfer learning according to claim 5, characterized in that: In S8, the relative majority voting model fusion strategy usually takes the category with the most votes as the prediction result and becomes the final classification category. If more than one category obtains the highest votes, one of them is randomly selected as the final classification category. The calculation formula is:
Citation Information
Cited By
Fruit image classification method based on shuttle-shaped dynamic neuron model
CN120014628A
Supermarket fruit image recognition system and method based on artificial intelligence algorithm
CN121074872A
Method for carrying out fruit type and position detection based on CSI (Channel State Information) channel data
CN121580003A
