Fruit surface defect detection method based on convolutional neural network

Abstract

The invention discloses a fruit surface defect detection method based on a convolutional neural network. The method comprises the following steps: acquiring fruit surface image data and performing data enhancement; preprocessing the fruit surface image data; performing image enhancement on the preprocessed fruit surface image data; dividing the image data set into a training set, a verification set and a test set; constructing a fruit image classification network model, and adding an SPP module and a CBAM module into the ResNet50 network model; training the fruit image classification network model based on the training set to obtain a trained fruit image classification network model; testing the fruit image classification network model based on the test set to obtain fruit surface defect classification accuracy; and obtaining a predicted fruit surface defect classification result based on the trained fruit image classification network model. The method can classify the fruits according to different quality grades, and has the advantages of good adaptability, high accuracy and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly relates to a method for detecting fruit surface defects based on a convolutional neural network. Background Art

[0002] With the continuous improvement of people's living standards and the continuous development of fruit cultivation techniques, the quality of fruits has received increasing attention. Whether a fruit has surface defects directly affects the classification of fruit grades. Judging by the naked eye not only has a certain degree of subjectivity but also has low efficiency.

[0003] Currently, the detection and grading of fruits still mostly rely on manual operations. Often due to the ambiguity of standards, the grading is inaccurate. A large number of errors are prone to occur in manual sorting or ordinary mechanical sorting equipment, resulting in uneven fruit quality; while existing fruit grading systems (such as a method for detecting fruit surface defects based on a convolutional neural network with the patent publication number CN112697984A) have deficiencies such as high cost, complex operation, and poor applicability, and have high requirements for production equipment and are difficult to use when computing resources are limited.

[0004] Therefore, there is an urgent need for a method for identifying fruit surface defects with low cost, good applicability, and high grading accuracy. Summary of the Invention

[0005] In order to overcome the defects and deficiencies existing in the prior art, the present invention provides a method for detecting fruit surface defects based on a convolutional neural network. During the process of classifying fruits after picking, the present invention can intelligently identify the defects on the fruit surface, and then classify the fruits according to different quality grades, and has the advantages of low cost, good adaptability, high accuracy, etc., expanding the application scenarios of deep learning technology.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention provides a method for detecting fruit surface defects based on a convolutional neural network, including the following steps:

[0008] Obtain fruit surface image data and perform data augmentation on the fruit surface image data;

[0009] Preprocess the fruit surface image data;

[0010] Perform image enhancement on the preprocessed fruit surface image data to improve the contrast between the fruit surface defect area and the normal area;

[0011] Divide the image data set into a training set, a validation set, and a test set;

[0012] Build a fruit image classification network model. Add an SPP module and a CBAM module to the ResNet50 network model. The SPP module performs pooling operations on the input image at different scales using pooling windows of different sizes, and splices all the pooled feature vectors to form a feature vector of a fixed size. The CBAM module adjusts the channel data feature weights based on spatial attention and channel attention.

[0013] Train the fruit image classification network model based on the training set to obtain the trained fruit image classification network model.

[0014] Test the fruit image classification network model based on the test set to obtain the classification accuracy of fruit surface defects.

[0015] Obtain the predicted classification result of fruit surface defects based on the trained fruit image classification network model.

[0016] As a preferred technical solution, perform data augmentation on the fruit surface image data, specifically including: data augmentation operations such as random scaling, inversion, cropping, rotation, and optical transformation.

[0017] As a preferred technical solution, perform preprocessing on the fruit surface image data, specifically including:

[0018] Perform noise reduction preprocessing on the fruit surface image data based on mean filtering. Given a template for the target pixel on the image, the template includes its surrounding neighboring pixels, and use the average value of all pixels in the template to replace the original pixel value.

[0019] As a preferred technical solution, the SPP module includes an input layer, multiple max-pooling layers, a splicing layer, and an output layer;

[0020] The input layer inputs the feature map. Each max-pooling layer performs pooling operations on the input feature map using pooling windows of different sizes, evenly divides the feature maps of different sizes into several grids of different sizes, performs pooling operations on the feature maps within each grid, the splicing layer splices all the pooled feature vectors, and the output layer outputs the spliced feature vector of a fixed size.

[0021] As a preferred technical solution, add a CBAM module after the max-pooling layer, after the second residual module, and after the fourth residual module in the ResNet50 network model.

[0022] The present invention also provides a fruit surface defect detection system based on a convolutional neural network, including: an image data acquisition module, a data augmentation module, a data preprocessing module, an image enhancement module, a data division module, a network model construction module, a network model training module, a network model testing module, and a defect classification result output module.

[0023] The image data acquisition module is used to acquire fruit surface image data;

[0024] The data augmentation module is used to perform data augmentation on the fruit surface image data;

[0025] The data preprocessing module is used to preprocess the fruit surface image data;

[0026] The image enhancement module is used to enhance the preprocessed fruit surface image data to improve the contrast between the defective area and the normal area on the fruit surface;

[0027] The data division module is used to divide the image dataset into a training set, a validation set, and a test set;

[0028] The network model construction module is used to construct a fruit image classification network model. An SPP module and a CBAM module are added to the ResNet50 network model. The SPP module performs pooling operations on the input image at different scales using pooling windows of different sizes, and splices all the pooled feature vectors to form a feature vector of a fixed size. The CBAM module adjusts the channel data feature weights based on spatial attention and channel attention;

[0029] The network model training module is used to train the fruit image classification network model based on the training set to obtain a trained fruit image classification network model;

[0030] The network model testing module is used to test the fruit image classification network model based on the test set to obtain the classification accuracy of fruit surface defects;

[0031] The defect classification result output module is used to obtain the predicted fruit surface defect classification result based on the trained fruit image classification network model.

[0032] As a preferred technical solution, the data augmentation module is used to perform data augmentation on the fruit surface image data, specifically including: data augmentation operations such as random scaling, inversion, cropping, rotation, and optical transformation.

[0033] As a preferred technical solution, the data preprocessing module is used to preprocess the fruit surface image data, specifically including:

[0034] Perform noise reduction preprocessing on the fruit surface image data based on mean filtering. Give a template to the target pixel on the image. The template includes its surrounding adjacent pixels, and use the average value of all pixels in the template to replace the original pixel value.

[0035] As a preferred technical solution, the SPP module includes an input layer, a plurality of max pooling layers, a splicing layer, and an output layer;

[0036] The input layer inputs a feature map. Each max pooling layer performs a pooling operation on the input feature map using pooling windows of different sizes, evenly divides the feature maps of different sizes into several grids of different sizes, and performs a pooling operation on the feature map within each grid. The splicing layer splices all the pooled feature vectors, and the output layer outputs the spliced feature vectors of a fixed size.

[0037] As a preferred technical solution, a CBAM module is added after the max pooling layer, after the second residual module, and after the fourth residual module in the ResNet50 network model.

[0038] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0039] (1) By introducing the SPP module into the ResNet50 network model, the present invention effectively avoids problems such as image distortion caused by cropping and scaling operations on the image area.

[0040] (2) By introducing the attention mechanism module CBAM into the ResNet50 network model, the present invention makes up for the lost information in average pooling to a certain extent.

[0041] (3) The fruit image classification network model of the present invention has a relatively high classification accuracy, reaching more than 95%, which can meet the requirements in actual production, has strong applicability, and can be applied in scenarios with limited computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a schematic flow chart of the fruit surface defect detection method based on a convolutional neural network of the present invention;

[0043] Figure 2 is a schematic network structure diagram of the SPP module of the present invention;

[0044] Figure 3 is a schematic network structure diagram of the CBAM module of the present invention;

[0045] Figure 4 is a schematic network structure diagram of the fruit image classification network model of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0047] Example 1

[0048] As Figure 1 shown, this embodiment provides a method for detecting fruit surface defects based on a convolutional neural network, including the following steps:

[0049] S1: Obtain fruit surface image data and perform data augmentation on the fruit surface image data;

[0050] In this embodiment, fruit surface images can be obtained online. Through various search engines and databases, relevant fruit surface image samples are collected, and attention should be paid to the balance of the fruit surface image samples to avoid a situation where the number of samples of different categories varies greatly;

[0051] In this embodiment, fruit surface images can also be obtained offline. The fruit is placed in a light box, and multiple high-precision cameras are used to collect images of the fruit from multiple angles to ensure that the image acquisition conditions for each fruit are the same;

[0052] In this embodiment, data augmentation operations such as random scaling, inversion, cropping, rotation, and optical transformation are performed on each fruit image to increase the number of the sample data set, enabling the network model to be fully trained and making the distribution of the sample data set more balanced, avoiding the problem that the model has a tendency during the training process due to an excessive number of samples of certain categories;

[0053] S2: Preprocess the image by means of mean filtering to exclude the interference of noise;

[0054] In this embodiment, the mean filtering method is used to perform noise reduction on the fruit image. A template is given to the target pixel on the image. The template includes its surrounding adjacent pixels (8 pixels surrounding the target pixel form a filtering template, that is, including the target pixel itself), and then the average value of all pixels in the template is used to replace the original pixel value;

[0055] S3: Through the method of image enhancement for the image after filtering processing, improve the contrast between the fruit surface defect area and the normal area, so that the network model can more accurately focus on the area of interest and improve the detection accuracy of the model;

[0056] Since the piecewise linear transformation is conceptually simple and intuitive, easy to implement, and it can be effectively used to enhance the contrast and adjust the brightness to improve the image quality, this embodiment uses the piecewise linear transformation method to make the defect area more prominent, improve the contrast between the defect area and the normal area, and enable the network model to more accurately focus on the area of interest;

[0057] S4: Divide the image data set into a training set, a validation set, and a test set;

[0058] In this embodiment, the fruit image dataset is divided into a training set, a validation set, and a test set according to a ratio of 3:1:1. In each dataset, the images are saved in different folders according to different categories. The classification labels of the fruit images include four labels: lesions, rot, mechanical damage, and fruit stalks.

[0059] S5: Construct a fruit image classification network model, adjust and optimize the structure of the ResNet50 network model, add an SPP module and a CBAM module to the network, which not only improves the applicability and classification accuracy of the model but also has little impact on the computational amount and training time.

[0060] As Figure 4 shown, in the fruit image classification network model, Input and Output represent the input feature map and the output feature map, Conv represents the convolutional layer, n×n represents the size of the convolutional kernel, BN represents batch normalization, Relu represents the use of the Relu activation function, MaxPool2d represents the max pooling layer, and the corresponding n×n represents the size of the pooling kernel. BottleNeck represents the residual block of ResNet50, ×n represents the number of layers, Avg represents the average pooling layer, and fc represents the fully connected layer. ResNet solves the degradation problem encountered in the training of deep networks by introducing a residual learning framework, and uses residual blocks (Residual Blocks) to greatly deepen the network depth and improve the training speed and detection accuracy of the network.

[0061] As Figure 2As shown, a Spatial Pyramid Pooling (SPP) module is added to the fruit image classification network model to avoid image distortion problems caused by cropping and scaling operations on image regions. An SPP module is added after the 7×7 input convolutional layer. In the figure, Input and Output represent the input feature map and the output feature map, MaxPool2d represents the max pooling layer, n×n represents the size of the pooling kernel, and Concat represents the concatenation operation. The SPP module effectively avoids problems such as image distortion caused by cropping and scaling operations on image regions, solves the problem of repetitive feature extraction of convolutional neural networks for images, saves computational costs, and the multi-scale features extracted using spatial pyramid pooling significantly enhance the network's feature extraction ability, which helps to improve the performance of the network model. It performs pooling operations on the input image at different scales using pooling windows of different sizes, and finally concatenates these pooling results to form a feature vector of a fixed size. Specifically, it evenly divides feature maps of different sizes into several grids of different sizes, performs pooling operations on the feature maps within each grid, and then connects all the pooled feature vectors as the input of the network. In this way, even if the input image sizes are different, feature vectors of a fixed length can be obtained for network training and inference.

[0062] As Figure 3 shown, an attention mechanism CBAM module is added to the fruit image classification network model, which makes up for the lost information in average pooling to a certain extent. In the figure, F represents the feature map, M represents the feature information, Conv layer represents the convolutional layer, MaxPool represents the max pooling layer, AvgPool represents the average pooling layer, MLP represents the multi-layer perceptron, vector represents the vector, Spatial Attention Module represents the spatial attention module, and Channel Attention Module represents the channel attention module. represents the addition operation, represents the multiplication operation, It represents normalizing the weights through the sigmoid activation function. CBAM starts from two scopes, namely the channel and the spatial scopes, introducing two analysis dimensions of spatial attention and channel attention to achieve an order attention structure from the channel to the space. The spatial attention can enable the neural network to pay more attention to the pixel regions that play a decisive role in classification in the image while ignoring the unimportant regions, and the channel attention is used to handle the allocation relationship of the feature map channels. At the same time, the attention allocation for the two dimensions enhances the improvement effect of the attention mechanism on the model performance. In the present invention, a CBAM module is added after the maximum pooling layer, after the second residual module, and after the fourth residual module, aiming to enhance the ability of the network to extract task-related regions, increase the weights of the channel data features with high correlation, and reduce the weights of the channel data features with low correlation, so as to obtain the optimal recognition ability and improve the recognition accuracy of the network.

[0063] S6: Set the training parameters, use the training set and the validation set to train and tune the convolutional neural network model, and obtain the network model with the best surface defect recognition effect;

[0064] In this embodiment, training parameters such as epoch, batch size, and learning rate are set, and the Adam optimizer is used. After a large number of trainings and debuggings, the network model with the best surface defect recognition effect is obtained.

[0065] In this embodiment, epoch is set to 300, batch size is set to 16, and the learning rate is set to 0.0002. And the cosine annealing algorithm is used to dynamically adjust the learning rate to avoid the oscillation phenomenon caused by too fast gradient descent during the training process, thereby improving the training stability and generalization ability of the model. Since the Adam optimizer contains the concept of momentum, it accumulates the exponentially decaying average of the previous gradients to help accelerate the learning. At the same time, it also uses the exponentially decaying average of the squared gradients to adaptively adjust the learning rate of each parameter, and has strong robustness and is widely used in deep learning tasks. Therefore, the Adam optimizer is used for training. After a large number of trainings and debuggings, the network model with the best surface defect recognition effect is obtained.

[0066] S7: Call the fruit image classification network model to conduct classification tests on the test set, use the classification accuracy as the model evaluation criterion to verify the model performance. By comparing the recognition result and the true category of the fruit defect, it can be detected whether the recognition method has the ability to recognize the fruit defect, and the accuracy of the surface defect recognition is output, thus completing the fruit surface defect recognition based on deep learning;

[0067] After obtaining a network model with a classification accuracy meeting the requirements in this embodiment, it can be deployed to the computer vision system of the fruit grader. After collecting fruit images using a camera, the images are used as the input of the network model, and the category of the fruit defect can be obtained in real time according to the output of the model. Then, the fruits are classified according to the set fruit grading standards, and the sorting work of the fruits can be realized in cooperation with the motion control module (such as a robotic arm), achieving the recognition of the surface defects of the fruits, which can effectively improve the efficiency and reduce the consumption of unnecessary human and material resources, effectively alleviating the difficulty of screening and classification in the case of a large number of fruits, and having the advantages of low cost, small implementation difficulty, strong applicability, and good detection effect.

[0068] Embodiment 2

[0069] This embodiment provides a fruit surface defect detection system based on a convolutional neural network for implementing the fruit surface defect detection method based on a convolutional neural network in the above Embodiment 1, including: an image data acquisition module, a data augmentation module, a data preprocessing module, an image enhancement module, a data division module, a network model construction module, a network model training module, a network model testing module, and a defect classification result output module;

[0070] In this embodiment, the image data acquisition module is used to acquire fruit surface image data;

[0071] In this embodiment, the data augmentation module is used to perform data augmentation on the fruit surface image data;

[0072] In this embodiment, the data preprocessing module is used to preprocess the fruit surface image data;

[0073] In this embodiment, the image enhancement module is used to perform image enhancement on the preprocessed fruit surface image data to improve the contrast between the fruit surface defect area and the normal area;

[0074] In this embodiment, the data division module is used to divide the image data set into a training set, a validation set, and a test set;

[0075] In this embodiment, the network model construction module is used to construct a fruit image classification network model, adding an SPP module and a CBAM module to the ResNet50 network model. The SPP module performs pooling operations on the input image at different scales using pooling windows of different sizes, and splices all the pooled feature vectors to form a feature vector of a fixed size. The CBAM module adjusts the channel data feature weights based on spatial attention and channel attention;

[0076] In this embodiment, the network model training module is used to train the fruit image classification network model based on the training set to obtain a trained fruit image classification network model;

[0077] In this embodiment, the network model testing module is used to test the fruit image classification network model based on a test set to obtain the classification accuracy of fruit surface defects.

[0078] In this embodiment, the defect classification result output module is used to obtain the predicted fruit surface defect classification result based on the trained fruit image classification network model.

[0079] In this embodiment, the data augmentation module is used to perform data augmentation on the fruit surface image data, specifically including: data augmentation operations such as random scaling, inversion, cropping, rotation, and optical transformation.

[0080] In this embodiment, the data preprocessing module is used to preprocess the fruit surface image data, specifically including:

[0081] Performing noise reduction preprocessing on the fruit surface image data based on mean filtering. Given a template for the target pixel on the image, the template includes its surrounding neighboring pixels, and the average value of all pixels in the template is used to replace the original pixel value.

[0082] In this embodiment, the SPP module includes an input layer, multiple max-pooling layers, a concatenation layer, and an output layer;

[0083] The input layer inputs a feature map. Each max-pooling layer performs a pooling operation on the input feature map using pooling windows of different sizes, evenly divides the feature maps of different sizes into several grids of different sizes, and performs a pooling operation on the feature map within each grid. The concatenation layer concatenates all the pooled feature vectors, and the output layer outputs the concatenated feature vector of a fixed size.

[0084] In this embodiment, a CBAM module is added after the max-pooling layer, after the second residual module, and after the fourth residual module in the ResNet50 network model.

[0085] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited by the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A method for detecting fruit surface defects based on a convolutional neural network, characterized in that, Including the following steps: Obtain the fruit surface image data and perform data augmentation on the fruit surface image data; Preprocess the fruit surface image data; Perform image enhancement on the preprocessed fruit surface image data to improve the contrast between the defective area and the normal area on the fruit surface; Divide the image dataset into a training set, a validation set, and a test set; Construct a fruit image classification network model. Add an SPP module and a CBAM module to the ResNet50 network model. The SPP module performs pooling operations on the input image at different scales using pooling windows of different sizes, and splices all the pooled feature vectors to form a feature vector of a fixed size. The CBAM module adjusts the channel data feature weights based on spatial attention and channel attention; Train the fruit image classification network model based on the training set to obtain the trained fruit image classification network model; Test the fruit image classification network model based on the test set to obtain the classification accuracy of the fruit surface defects; Obtain the predicted fruit surface defect classification result based on the trained fruit image classification network model.

2. The method for detecting fruit surface defects based on a convolutional neural network according to claim 1, wherein Perform data augmentation on the fruit surface image data, specifically including: data augmentation operations such as random scaling, inversion, cropping, rotation, and optical transformation.

3. The fruit surface defect detection method based on a convolutional neural network according to claim 1, wherein Preprocess the fruit surface image data, specifically including: Perform noise reduction preprocessing on the fruit surface image data based on mean filtering. Give a template to the target pixel on the image. The template includes its surrounding adjacent pixels, and use the average value of all the pixels in the template to replace the original pixel value.

4. The method for detecting fruit surface defects based on a convolutional neural network according to claim 1, characterized in that, The SPP module includes an input layer, multiple max pooling layers, a splicing layer, and an output layer; The input layer inputs the feature map. Each max pooling layer performs pooling operations on the input feature map using pooling windows of different sizes, evenly divides the feature maps of different sizes into several grids of different sizes, and performs pooling operations on the feature maps within each grid. The splicing layer splices all the pooled feature vectors, and the output layer outputs the feature vector of the fixed size after splicing.

5. The fruit surface defect detection method based on a convolutional neural network according to claim 1, characterized in that, Add a CBAM module after the max pooling layer, after the second residual module, and after the fourth residual module in the ResNet50 network model.

6. A fruit surface defect detection system based on a convolutional neural network, characterized in that, Including: An image data acquisition module, a data augmentation module, a data preprocessing module, an image enhancement module, a data division module, a network model construction module, a network model training module, a network model testing module, and a defect classification result output module; The image data acquisition module is used to obtain the fruit surface image data; The data augmentation module is used to perform data augmentation on the fruit surface image data; The data preprocessing module is used to preprocess the fruit surface image data; The image enhancement module is used to perform image enhancement on the preprocessed fruit surface image data to improve the contrast between the defective area and the normal area on the fruit surface; The data division module is used to divide the image dataset into a training set, a validation set, and a test set; The network model construction module is used to construct a fruit image classification network model. An SPP module and a CBAM module are added to the ResNet50 network model. The SPP module performs pooling operations on the input image at different scales using pooling windows of different sizes, and splices all the pooled feature vectors to form a feature vector of a fixed size. The CBAM module adjusts the feature weights of channel data based on spatial attention and channel attention; The network model training module is used to train the fruit image classification network model based on the training set to obtain the trained fruit image classification network model; The network model testing module is used to test the fruit image classification network model based on the test set to obtain the classification accuracy of fruit surface defects; The defect classification result output module is used to obtain the predicted fruit surface defect classification result based on the trained fruit image classification network model.

7. The fruit surface defect detection system based on a convolutional neural network according to claim 6, wherein The data augmentation module is used to perform data augmentation on the fruit surface image data, specifically including: data augmentation operations such as random scaling, inversion, cropping, rotation, and optical transformation.

8. The method for detecting fruit surface defects based on a convolutional neural network according to claim 6, wherein The data preprocessing module is used to preprocess the fruit surface image data, specifically including: Performing noise reduction preprocessing on the fruit surface image data based on mean filtering. A template is given to the target pixel on the image, and the template includes its surrounding adjacent pixels. The average value of all the pixels in the template is used to replace the original pixel value.

9. The method for detecting fruit surface defects based on a convolutional neural network according to claim 6, wherein, The SPP module includes an input layer, multiple max pooling layers, a splicing layer, and an output layer; The input layer inputs the feature map. Each max pooling layer performs a pooling operation on the input feature map using a pooling window of a different size, evenly divides the feature maps of different sizes into several grids of different sizes, performs a pooling operation on the feature map within each grid. The splicing layer splices all the pooled feature vectors, and the output layer outputs the spliced feature vector of a fixed size.

10. The method for detecting fruit surface defects based on a convolutional neural network according to claim 6, characterized in that, A CBAM module is added after the max pooling layer, after the second residual module, and after the fourth residual module in the ResNet50 network model.

Images