A fruit variety recognition method based on parallel convolutional neural network

Abstract

The invention discloses a fruit variety recognition method based on a parallel convolutional neural network, comprising step 1: changing a fruit image with a category label into a 3-channel image with a pixel of 128×128; step 2: performing translation, rotation, and mirror flipping operations on the result of step 1 at the same time to generate a fruit image data set of various scales; step 3: inputting the result of step 2 into a generative adversarial network model to enhance the data; step 4: constructing a parallel convolutional neural network model, and performing multi-scale feature extraction on the result of step 3; step 5: according to the features extracted in step 4, the parallel convolutional neural network model predicts the category of the fruit image and compares it with the category label, and the parallel convolutional neural network model is trained according to the comparison result. The invention can improve the accuracy of fruit variety recognition based on images, and has important significance in the mechanization and intelligent application of the fruit industry.

Description

Technical Field

[0001] The invention relates to image recognition, and in particular to a fruit variety recognition method based on a parallel convolutional neural network. Background Art

[0002] At present, the mechanization level in the current fruit industry structure in my country is low, and most of the production links, especially fruit picking, mainly rely on time-consuming and labor-intensive manual work. The entire fruit production operation includes picking, storage, transportation, processing and sales. Therefore, the research and development of fruit agricultural production robots is an inevitable trend to improve fruit production efficiency and save labor costs. Whether it is a picking and sorting robot, or a fruit quality and variety detection system in the fruit production link, their normal operation depends on the correct recognition of the fruit by the image processing module. For example, the picking robot can only provide motion parameters for the mechanical arm and complete the fruit picking operation after it recognizes the fruit from the fruit tree and obtains the accurate position of the fruit.

[0003] In recent years, deep learning technology has developed rapidly. It can excellently complete various computer vision tasks and has been gradually applied to the agricultural field. The constructed deep learning model can automatically learn the characteristic information of different things and obtain the differences between categories after a large amount of data training. The deep learning model can convert the original data into a more abstract and advanced expression after training and learning, and then complete tasks such as image classification and detection. However, the accuracy of the current fruit variety recognition method based on image deep learning is low and cannot fully meet the requirements of practical applications. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a fruit variety recognition method based on parallel convolutional neural network, which overcomes the problem of small existing data sets and low recognition rate of traditional parallel convolutional neural network, and realizes fast and accurate distinction between similar fruit varieties.

[0005] Technical solution: The present invention provides a fruit variety identification method based on a parallel convolutional neural network, comprising the following steps:

[0006] Step 1: Change the fruit image with category labels into a 3-channel image with a pixel size of 128×128;

[0007] Step 2: Perform translation, rotation, and mirror flip operations on the results of step 1 to generate fruit image datasets of various scales;

[0008] Step 3: Input the result of step 2 into the generative adversarial network model to enhance the data;

[0009] Step 4: Build a parallel convolutional neural network model and perform multi-scale feature extraction on the results of step 3;

[0010] Step 5: Based on the features extracted in step 4, the parallel convolutional neural network model predicts the category of the fruit image and compares it with the category label. The parallel convolutional neural network model is trained based on the comparison results to maximize the accuracy of the training model and minimize the loss rate.

[0011] Preferably, step 2 includes rotating the image by 30°, 60°, 90°, translating by 10%, 20%, 30%, and mirroring by 30°, 60°, 90°.

[0012] Among them, the generative adversarial network in step 3 includes a generator and a discriminator. Noise is input into the generator, and then random sampling is performed from the noise. Through a 5-layer network, a 128×128 3-channel data sample is generated; the discriminator compares the data sample with the real sample to determine the authenticity of the sample generated by the generator. The generator is updated with the help of the fixed parameters of the discriminator to generate pictures that are more difficult for the discriminator to distinguish the authenticity.

[0013] The parallel convolutional neural network used in step 4 includes 8 convolutional layers, 6 maximum pooling layers, and finally a fully connected layer. The fully connected layer integrates the local information with classification information of the convolutional layer and the maximum pooling layer.

[0014] Step 5 also includes optimizing the convolutional neural network model by combining the maximum class distance loss function with the SoftmaxWithLoss loss function.

[0015] Take the fruit image to be classified as the target image, perform steps 1 to 3, and then input the trained parallel convolutional neural network model for fruit recognition.

[0016] Traditional data enhancement and generative adversarial network are used to perform data enhancement; and the fruit images to be classified are brought into the trained model for training.

[0017] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: using a generative adversarial network combined with traditional data enhancement methods to generate a large number of high-quality data sets; using a parallel convolutional neural network to complete the simultaneous extraction of features of different scales, making the feature expression richer and allowing the network to extract more feature information; using the maximum class distance loss function and SoftmaxWithLoss to increase the distance between similar varieties, thereby improving the recognition accuracy between similar varieties. The present invention can achieve an accuracy of 98.85% on the public data set fruit-360. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of the present invention;

[0019] Figure 2 A schematic diagram of a generative adversarial network of the present invention;

[0020] Figure 3 Schematic diagram of the parallel convolutional neural network of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is further described below in conjunction with the accompanying drawings. The present invention provides a method for training a fruit variety classification model, comprising the following steps:

[0022] Step 1: Perform traditional data augmentation on fruit images with category labels to form fruit images with multiple datasets.

[0023] Step 2: Build a generative adversarial network model and use it to generate more high-quality data sets. This network includes a discriminator and a generator, which train the generative adversarial network model through a game. The discriminator adjusts its own parameters through the parameters passed by the generator to determine the authenticity of the incoming data. The generator updates its own parameters with the help of the fixed parameters of the discriminator to generate more data that is difficult for the discriminator to judge. Finally, the discriminator and the generator reach a balance point and the training ends.

[0024] When training the generator, the parameters of the discriminator are fixed. The generator uses the fixed parameters of the discriminator to update its own parameters so that it is as difficult as possible for the discriminator to determine whether the generated samples are real data or simulated samples.

[0025] When training the discriminator, the parameters of the generator are fixed and the data generated by the generator are treated as negative samples. The collection of real data is input into the discriminator as positive samples. Through the parameters passed by the generator, the discriminator adjusts its own parameters to make it judge the source of the incoming data as accurately as possible.

[0026] Step 3: Use the fruit images with category labels as training data and bring them into the parallel convolutional neural network model for feature extraction. C1 and C2 are 16 3×3 convolution kernels, using all zero padding, step size 1, and output 128×128×3. S1 is a 2×2 maximum pooling layer with a step size of 2 and an output of 64×64×16. C3 is 16 3×3 convolution kernels, using all zero padding, step size 1, and output 64×64×16. C4 is 16 3×3 convolution kernels, using all zero padding, step size 1, and output 64×64×16. S2 is a 2×2 maximum pooling layer with a step size of 2 and an output of 32×32×16. The generated feature image is input into two parallel channels a and b. Channel a contains 32 3×3 convolution layers and a 2×2 maximum pooling layer. The convolution layer uses full zero padding with a step size of 1 and the pooling layer has a step size of 4. Channel b contains 32 5×5 convolution kernels and two 2×2 maximum pooling layers. The convolution layer uses full zero padding with a step size of 1 and the maximum pooling layer has a step size of 2. Channels a and b each generate 32 8×8 feature maps. Then the 64 8×8 feature maps generated by the two channels are used as the input of C5. C5 and C6 are 64 convolution layers of size 3×3, using full zero padding with a step size of 1. S3 is a maximum pooling layer of size 2×2 with a step size of 2. Finally, it is a fully connected layer to integrate the local information with classification information of the convolution layer and the pooling layer.

[0027] Step 4: Use the extracted feature information to predict the variety category of the fruit image and compare it with the corresponding label, and train the parallel convolutional neural network model based on the comparison results.

[0028] In step 4, the maximum class distance loss function between different varieties of the same category can be used to predict the maximum class distance loss function and the SoftmaxWithLoss function to form a new loss function to optimize the convolutional neural network. The maximum class distance loss function formula is as follows:

[0029]

[0030]

[0031] Among them, i represents the i-th type of fruit, j represents the j-th type of fruit, m represents the total number of fruit types, M (i) represents the mean of the i-th type of fruit, M (j) represents the mean of the jth class, n represents the number of samples of the i-th fruit, x (i,e) represents the value of the e-th variety of the i-th category of fruit, h w,b (x (i,e) ) indicates wx (i,e-1)+b, w represents the weight of the e-th variety, and b represents the bias term. The identified variety does not need to be compared one by one with the rest of the fruit, but only needs to be compared with similar categories. The maximum class distance will increase the distance between similar varieties as the number of training times increases. Combining the maximum class distance with SoftmaxWithLoss, we get the formula:

[0032] J=S-λL (3)

[0033] Among them, S represents SoftmaxWithLoss, L represents the maximum class spacing function, λ represents the hyperparameter, and J makes the classification between similar classes more and more accurate. Taking the derivative of L, we can get the formula:

[0034]

[0035] Where Z represents f(h w,b (x (i,e) )), f, l, and n represent the activation function, the number of convolutional layers, and the number of samples, respectively.

[0036] In step 3, the last pooling layer features and the last fully connected layer features of each image can also be extracted; the original image can also be regularized, the cut image can be subjected to maximum pooling, and then regularized.

[0037] In step 4, the Softmax classifier can also be used for variety prediction.

[0038] The method described in the present invention is also effective for fruit images without internal annotations in the images. Internal annotations refer to: bounding-box annotations, contour annotations, etc.

Claims

1. A fruit variety identification method based on parallel convolutional neural network, characterized in that: The steps include: Step 1: Change the fruit image with category labels into a 3-channel image with a pixel size of 128×128; Step 2: The result of step 1 is simultaneously translated, rotated, and mirrored to generate a fruit image dataset of various scales; including rotating the image by 30°, 60°, and 90°, translating by 10%, 20%, and 30%, and mirroring by 30°, 60°, and 90°; Step 3: Input the result of step 2 into the generative adversarial network model to enhance the data; The generative adversarial network includes a generator and a discriminator. Noise is input into the generator, and then random sampling is performed from the noise. Through a 5-layer network, a 128×128 3-channel data sample is generated. The discriminator compares the data sample with the real sample to determine the authenticity of the sample generated by the generator. The generator is updated with the help of the fixed parameters of the discriminator to generate pictures that are more difficult for the discriminator to distinguish between true and false. Extract the last pooling layer features and the last fully connected layer features of each image; You can also perform regularization on the original image, perform maximum pooling on the cut image, and then perform regularization again; Step 4: construct a parallel convolutional neural network model to perform multi-scale feature extraction on the result of step 3; the parallel convolutional neural network used includes 6 convolutional layers, 3 maximum pooling layers and a fully connected layer, and the fully connected layer integrates the local information with classification information of the convolutional layer and the maximum pooling layer; The details are as follows: C1 and C2 are 16 3×3 convolution kernels, using all zero padding, with a step size of 1, and the output is 128×128×3; S1 is a 2×2 maximum pooling layer, with a step size of 2, and the output is 64×64×16; C3 is 16 3×3 convolution kernels, using all zero padding, with a step size of 1, and the output is 64×64×16; C4 is 16 3×3 convolution kernels, using all zero padding, with a step size of 1, and the output is 64×64×16, S2 is a 2×2 maximum pooling layer, with a step size of 2, and the output is 32×32×16; the generated feature image is input into two parallel channels a and b. Channel a contains 32 3×3 convolution layers and a 2×2 maximum pooling layer. The convolution layer uses all zero padding, with a step size of 1, and the pooling layer has a step size of 4; channel b The channel contains 32 5×5 convolution kernels and two 2×2 maximum pooling layers, where the convolution layer uses full zero padding with a step size of 1, and the maximum pooling layer has a step size of 2. The a and b channels each generate 32 8×8 feature maps, and then the 64 8×8 feature maps generated by the two channels are used as the input of C5. C5 and C6 are 64 convolution layers of size 3×3, using full zero padding with a step size of 1. S3 is a maximum pooling layer of size 2×2 with a step size of 2; the last is a fully connected layer, which integrates the local information with classification information of the convolution layer and the pooling layer; the details are as follows: the maximum class distance loss function between different varieties of the same category is used to predict the maximum class distance loss function and the SoftmaxWithLoss function to form a new loss function to optimize the convolutional neural network; the maximum class distance loss function formula is as follows: Among them, i represents the i-th type of fruit, j represents the j-th type of fruit, m represents the total number of fruit types, M (i) represents the mean of the i-th type of fruit, M (j) represents the mean of the jth class, n represents the number of samples of the i-th fruit, x (i,e) represents the value of the e-th variety of the i-th category of fruit, h w,b (x (i,e) ) indicates wx (i,e-1) +b, w represents the weight of the e-th variety, and b represents the bias term; the identified variety does not need to be compared one by one with the rest of the fruit, but only needs to be compared with similar categories. The maximum class distance will increase the distance between similar varieties as the number of training times increases; the maximum class distance is combined with SoftmaxWithLoss to obtain the formula: J=S-λL (3) Among them, S represents SoftmaxWithLoss, L represents the maximum class spacing function, λ represents the hyperparameter, and J makes the classification between similar classes more and more accurate; taking the derivative of L, we can get the formula: Where Z represents f(h w,b (x (i,e) ), f, l, and n represent the activation function, the number of convolutional layers, and the number of samples, respectively; Step 5: Based on the features extracted in step 4, the parallel convolutional neural network model predicts the category of the fruit image and compares it with the category label, and trains the parallel convolutional neural network model based on the comparison results to improve the recognition accuracy.

2. A fruit variety identification method based on parallel convolutional neural network according to claim 1, characterized in that: The method also includes taking the image of the fruit to be classified as the target image, performing steps 1 to 3, and then inputting the trained parallel convolutional neural network model to identify the fruit variety.

Images