Fruit and vegetable grading method based on deep learning model

Abstract

The invention discloses a fruit and vegetable grading method based on a deep learning model, which is used for gradually judging the quality grade of fruits and vegetables through staged processing. According to the sectional type grading strategy, on one hand, besides characteristic values of fruits and vegetables, the fruit and vegetable defect types of fruit and vegetable images are recognized through a deep learning semantic segmentation network model, the fruit and vegetable defect types are introduced to serve as grading bases, the diversity of judgment factors is remarkably improved, and the grading result is finer and more accurate; and on the other hand, through segmented processing, fruits and vegetables without fruit and vegetable defects can be rapidly screened in the early stage, the fruit and vegetable defect types are preferentially utilized for classification, unnecessary characteristic value calculation is avoided, the calculation amount is greatly reduced, and the classification efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic fruit and vegetable grading, and particularly relates to a fruit and vegetable grading method based on a deep learning model. BACKGROUND

[0002] In recent years, many scholars at home and abroad have researched the agricultural product quality detection method based on machine vision. However, the fruit and vegetable surface quality grading by using the traditional machine learning method has problems such as the need for human detailed description knowledge assistance, the interference of human judgment factors, and the like, and has disadvantages such as long calculation time and low accuracy in processing a large amount of data.

[0003] With the development of science and technology, the data model deep learning gradually rises and gradually applies to image recognition. The complex network structure and data training model not only reduce the workload, but also improve the detection speed and accuracy. The fruit and vegetable grading based on the deep learning model mainly includes two work contents: model training and learning and fruit and vegetable recognition grading.

[0004] In the model training, the labeling of the training sample is one of the main work contents, and directly determines the accuracy of the model after training. In the fruit and vegetable grading, the labeling work of the existing model training is basically completed by artificial, which is very time-consuming and laborious, especially for some fruits and vegetables such as gold orange and pear, in addition to the fruit defects such as scab and bruise, the fruit stem and a large number of black spots are also distributed. No matter it is the fruit stem or the black spot, it is easy to be misjudged as the fruit defect in the labeling, and in order to achieve high recognition accuracy, a large number of black spot samples are also needed in the training model. Therefore, the labeling steps need to be distinguished and labeled point by point, which greatly increases the labeling workload and difficulty, and makes the labeling work very strong, which is eye-straining and laborious, and the labeling efficiency is low, and the labeling accuracy is also low, which affects the result of the model training.

[0005] In the fruit and vegetable recognition grading, the existing conventional method is to use the model to detect and recognize or calculate the quality characteristics of the fruit and vegetable, such as the volume, shape, color, and surface fruit defect, and then grade once according to these characteristic values. In this grading method, in addition to the surface fruit defect, each quality characteristic of the fruit and vegetable, such as the volume, shape, color, and the like, needs to be recognized or calculated, and the calculation amount is very large, especially when the quality characteristics of the fruit and vegetable are more, the calculation amount is multiplied, and the huge calculation amount seriously affects the grading efficiency.

[0006] In summary, the existing fruit and vegetable grading method based on the deep learning model has heavy tasks in the model training and the model recognition grading, and the grading efficiency is low, which needs to be further improved. SUMMARY

[0007] The application aims to provide a fruit and vegetable grading method based on a deep learning model.

[0008] The technical solution for achieving the application is: a fruit and vegetable grading method based on a deep learning model, characterized by comprising the following steps: C1. Real-time collection of fruit and vegetable images c; C2. Preprocessing of the fruit and vegetable images c collected in real time in step C1, the preprocessing operation including image segmentation processing: extracting the fruit and vegetable part of interest in the image, and removing the background part useless for image recognition; C3. Cutting the image after the preprocessing in step C2 into small pieces to obtain a plurality of small piece cut images c; C4. Using a neural network model to recognize each cut image c obtained in step C4; C5. According to the recognition result of step C4, judging whether there is one or more cut images c with fruit and vegetable defects in the cut images c: If all the cut images c are determined in step C5 to have no fruit and vegetable defects, the characteristic value of the fruit and vegetable in the fruit and vegetable image c collected in real time in step C1 is calculated, and the fruit and vegetable is graded according to the characteristic value of the fruit and vegetable; the characteristic value of the fruit and vegetable includes one or more values of the area proportion of each color of the fruit and vegetable, the fruit shape, and the volume; If one or more images have fruit and vegetable defects in step C5, a semantic segmentation network model is used to recognize the fruit and vegetable defects of the fruit and vegetable image c after the preprocessing in step C2, and the fruit and vegetable defect type is judged; after the fruit and vegetable defect type is determined, the fruit and vegetable is first graded according to the fruit and vegetable defect type; when the fruit and vegetable cannot be graded by the fruit and vegetable defect type, the characteristic value of the fruit and vegetable in the fruit and vegetable image c collected in real time in step C1 is calculated, and the fruit and vegetable is graded according to the characteristic value of the fruit and vegetable.

[0009] Further, the neural network model in step C4 is a deep learning neural network model obtained after training and learning; the steps of training and learning the neural network model include: A1. Batch collection of fruit and vegetable images a; A2. Sample labeling; the specific steps include: A2.1. Preprocessing of the fruit and vegetable images a obtained in step A1, the preprocessing operation including image segmentation processing: extracting the fruit and vegetable part of interest in the image, and removing the background part useless for image recognition; A2.2. Cutting the image after the preprocessing in step A2.1 into small pieces to obtain a plurality of small piece cut images a, and dividing the cut images a into training set samples; A2.3. Overall labeling of the cut images a of the training set samples divided in step A2.2, wherein: The cut image a of the fruitless vegetable with no defect and no black spot is labeled as the first category; The cut image a of the fruit-bearing vegetable with defect is labeled as the second category; The cut image a of the fruitless vegetable with no defect but with black spot is labeled as the third category; The image with more than one color block showing gray is the image with black spot, otherwise, the image without any color block showing gray is the image without black spot; the image with any one or more of the fruit vegetable defects including scab, wrinkled skin, stab injury and crush injury is the image with fruit vegetable defect, otherwise, the image without any fruit vegetable defect including scab, wrinkled skin, stab injury and crush injury is the image without fruit vegetable defect; A3. Neural network model training and learning; the specific steps include: A3.1. Select a neural network model; A3.2. Use the cut image a of the labeled training set sample in step A2.3 to train the neural network model determined in step A3.1; finally obtain the deep learning neural network model.

[0010] Further, in the training and learning of the neural network model, the step of cutting the image after pre-processing in step A2.1 into small pieces in step A2.2 is to take the center point of the image as the center to perform class sector segmentation on the image after pre-processing in step A2.1.

[0011] Further, in step A2.2, the cut image a is divided into training set samples, validation set samples and test set samples; in step A2.3, the cut image a of the training set samples and the validation set samples in step A2.2 is annotated as a whole; step A2.3 is to use the cut image a of the labeled training set sample in step A2.3 to train the neural network model determined in step A3.1; step A3.2 is to use the cut image a of the labeled validation set sample in step A2.3 to test the neural network model in training, adjust and optimize the parameters of the neural network model; use the cut image a of the test set sample in step A2.2 to evaluate the trained neural network model; finally obtain the deep learning neural network model.

[0012] Further, the semantic segmentation network model in step C5 is the deep learning semantic segmentation network model obtained after training and learning; the steps of semantic segmentation network model training and learning include: B1. Batch collect fruit vegetable images b; B2. Sample annotation; the specific steps include: B2.1. Pre-process the fruit vegetable images b collected in step B1; the pre-processed fruit vegetable images b are divided into training set samples; B2.2. Classify and label the fruit and vegetable defects of the fruit and vegetable image b in the training set sample preprocessed in step B2.1, and classify the fruit and vegetable defects into scab, wrinkled skin, stab injury or crush injury; B3. Training and learning of the semantic segmentation network model; the specific steps include: B3.1. Select a semantic segmentation network model; B3.2. Train the semantic segmentation network model determined in step B3.1 using the fruit and vegetable image b of the training set sample labeled in step B2.2; and finally obtain the semantic segmentation network model of the deep learning.

[0013] Further, in step B2.1, the preprocessed fruit and vegetable image b is divided into training set samples, validation set samples and test set samples; step B2.2 is to classify and label the fruit and vegetable defects of the fruit and vegetable image b in the training set sample and the validation set sample preprocessed in step B2.1; step B3.2 is to train the semantic segmentation network model determined in step B3.1 using the fruit and vegetable image b of the training set sample labeled in step B2.2; to test the trained semantic segmentation network model using the fruit and vegetable image b of the validation set sample labeled in step B2.2, to adjust and optimize the parameters of the semantic segmentation network model; to evaluate the trained semantic segmentation network model using the fruit and vegetable image b of the test set sample in step B2.1; and finally to obtain the semantic segmentation network model of the deep learning model.

[0014] Further, in step B2.1, the preprocessing operation includes image segmentation processing, extracting the fruit and vegetable part of interest in the image, and removing the background part useless for image recognition; in steps A2.1, B2.1 and C2, the preprocessing operation further includes binaryzation processing and noise reduction processing, and the image segmentation processing, binaryzation processing and noise reduction processing are sequentially performed in turn.

[0015] Further, in step C5, the feature value of the fruit and vegetable includes the area ratio of each color of the fruit and vegetable, and the steps of calculating the area ratio of each color of the fruit and vegetable are as follows: converting the fruit and vegetable image c collected in step C1 from the RGB color space to the HSV color space; defining the hue range of each color according to the characteristics of the HSV color space; performing threshold segmentation on the image using the defined color range to extract the pixel region of each color; and calculating the area ratio of each color. For example, the area ratio values of red, orange and yellow are calculated to analyze the maturity or quality of the kumquat.

[0016] Further, in step C5, the characteristic value of the fruit and vegetable includes the fruit and vegetable shape, and the step of calculating the fruit and vegetable shape is: performing gray-scale processing on the fruit and vegetable image c collected in real time in step C1, and converting it into a gray-scale image; using an image segmentation technique to extract the contour of the fruit and vegetable, the image segmentation technique being any one of threshold segmentation or edge detection; using an ellipse fitting algorithm to fit the contour of the fruit and vegetable, and obtaining the parameters of the ellipse; and determining the fruit shape of the fruit and vegetable according to the parameters of the ellipse. The ellipse fitting algorithm is, for example, the least square method. When the fruit shape of the fruit and vegetable is determined according to the parameters of the ellipse, a fruit shape index can be defined as the ratio of the major axis of the ellipse to the minor axis of the ellipse, and then the fruit shape category of the fruit and vegetable can be determined according to the value of the fruit shape index: if the fruit shape index > 1.0, the fruit and vegetable is long circular; if 0.9 ≤ fruit shape index < 1.0, the fruit and vegetable is conical or elliptical; if 0.8 ≤ fruit shape index < 0.9, the fruit and vegetable is circular or nearly circular; and if 0.6 ≤ fruit shape index < 0.8, the fruit and vegetable is flat circular.

[0017] Further, in step C5, the characteristic value of the fruit and vegetable includes the fruit and vegetable volume, and the step of calculating the fruit and vegetable volume is: performing gray-scale processing on the fruit and vegetable image c collected in real time in step C1, and converting it into a gray-scale image; using an image segmentation technique to extract the contour of the fruit and vegetable, the image segmentation technique being any one of threshold segmentation or edge detection; using an ellipse fitting algorithm to fit the contour of the fruit and vegetable, and obtaining the parameters of the ellipse; and calculating the volume of the fruit and vegetable according to the parameters of the ellipse. According to international standards, the fruit and vegetable is divided into extra-large fruit, large fruit, medium fruit, small fruit, and extra-small fruit.

[0018] Further, in step C5, the characteristic value of the fruit and vegetable includes the area ratio of each color of the fruit and vegetable, the fruit shape, and the volume; and the defect type of the fruit and vegetable is divided into scab, wrinkled skin, stab injury, and crush injury.

[0019] Further, in step C5, according to the recognition result of step C4, it is determined whether there is one or more than one fruit and vegetable defect in the cut image c. If all the cut images c are determined to have no defects in step C5, the characteristic value of the fruit or vegetable in the fruit or vegetable image c collected in real time in step C1 is calculated, and the fruit or vegetable is graded according to the characteristic value of the fruit or vegetable: if the proportion of the area of yellow in each color of the fruit or vegetable is less than N%, the ratio of the long axis to the short axis of the fruit or vegetable is greater than M%, and the volume of the fruit or vegetable is any one of large fruit or extra large fruit, the fruit or vegetable is determined to be a special grade fruit; if the proportion of the area of yellow in each color of the fruit or vegetable is less than N%, the ratio of the long axis to the short axis of the fruit or vegetable is greater than M%, and the volume of the fruit or vegetable is any one of medium fruit, small fruit or extra small fruit, the fruit or vegetable is determined to be a first grade fruit; if the proportion of the area of yellow in each color of the fruit or vegetable is less than N%, the ratio of the long axis to the short axis of the fruit or vegetable is greater than M%, and the volume of the fruit or vegetable is medium fruit, the fruit or vegetable is determined to be a second grade fruit; if the proportion of the area of yellow in each color of the fruit or vegetable is less than N%, the ratio of the long axis to the short axis of the fruit or vegetable is greater than M%, and the volume of the fruit or vegetable is small fruit, the fruit or vegetable is determined to be a third grade fruit; if the proportion of the area of yellow in each color of the fruit or vegetable is less than N%, the ratio of the long axis to the short axis of the fruit or vegetable is greater than M%, and the volume of the fruit or vegetable is extra small fruit, the fruit or vegetable is determined to be a fourth grade fruit; If any one of the cut images c has defects in step C5, the fruit or vegetable defects in the fruit or vegetable image c collected in real time in step C1 are identified using the deep learning semantic segmentation network model in step B3.2, and the type of the fruit or vegetable defects is determined. After determining the type of the fruit or vegetable defects, the fruit or vegetable is first graded according to the type of the fruit or vegetable defects: if the type of the fruit or vegetable defects includes bruising, the fruit or vegetable is determined to be a fourth grade fruit; if the type of the fruit or vegetable defects does not include bruising, but includes any one of puncture or wrinkling, the fruit or vegetable is determined to be a third grade fruit; if the type of the fruit or vegetable defects does not include bruising, puncture and wrinkling, the characteristic value of the fruit or vegetable in the fruit or vegetable image c collected in real time in step C1 is calculated, and the fruit or vegetable is graded according to the characteristic value of the fruit or vegetable: if the volume of the fruit or vegetable is any one of small fruit or extra small fruit, the fruit or vegetable is determined to be a third grade fruit; if the volume of the fruit or vegetable is any one of medium fruit, large fruit or extra large fruit, but the proportion of the area of yellow in each color of the fruit or vegetable is less than N% and the ratio of the long axis to the short axis of the fruit or vegetable is greater than M%, the fruit or vegetable is determined to be a second grade fruit; if the volume of the fruit or vegetable is any one of medium fruit, large fruit or extra large fruit, the proportion of the area of yellow in each color of the fruit or vegetable is less than N%, and the ratio of the long axis to the short axis of the fruit or vegetable is greater than M%, the fruit or vegetable is determined to be a first grade fruit; N and M are natural numbers greater than 0 and less than 100.

[0020] The fruit and vegetable grading method based on the deep learning model of the present application, compared with the fruit and vegetable grading method of the traditional one-time identification of fruit and vegetable defects and characteristic values, and grading according to the same, gradually judges the quality grade of fruit and vegetable through staged processing. This kind of segmented grading strategy, on the one hand, in addition to the characteristic value of fruit and vegetable, it also identifies the fruit and vegetable defect type of the fruit and vegetable image by using the semantic segmentation network model of deep learning, introduces the fruit and vegetable defect type as the grading basis, significantly increases the diversity of judgment factors, makes the grading result more fine and accurate; on the other hand, through segmented processing, the present application can quickly screen fruit and vegetable without defects in the early stage, and preferentially use fruit and vegetable defect type for grading, avoiding unnecessary characteristic value calculation, thereby greatly reducing the calculation amount and improving the grading efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is the flow chart of the training and learning of the neural network model of the present application; Figure 2 is the flow chart of the training and learning of the semantic segmentation network model of the present application; Figure 3 is the flow chart of the fruit and vegetable grading method based on the deep learning model of the present application; Figure 4 is the original image of the fruit and vegetable image a in the training and learning of the neural network model of the present application; Figure 5 is Figure 4 the image obtained after image segmentation processing; Figure 6 is Figure 5 the image obtained after binaryzation processing; Figure 7 is Figure 6 the image obtained after noise reduction processing; Figure 8 is Figure 4 the final image obtained after pretreatment; Figure 9 is the cut block image a obtained after the pretreated image in the training and learning of the neural network model of the present application is cut in matrix form; Figure 10 is the comparison chart of the identification accuracy of ResNet50, VGG16 and AlexNet three kinds of models under different cut block quantities in the training and learning of the neural network model of the present application; Figure 11 is the schematic diagram of the class fan-shaped segmentation of the pretreated image in the training and learning of the neural network model of the present application; Figure 12 is the single block cut block image a obtained after the pretreated image in the training and learning of the neural network model of the present application is segmented in a class fan-shaped manner; Figure 13 is a fruit and vegetable image a after fruit and vegetable defect type annotation in the fruit and vegetable grading method based on the deep learning model of the present application; Figure 14 is a label image generated from the fruit and vegetable image a after annotation in the fruit and vegetable grading method based on the deep learning model of the present application; Figure 15 is a scabbed fruit and vegetable image and a recognition image after detection using the semantic segmentation network model of the present application; Figure 16 is a wrinkled fruit and vegetable image and a recognition image after detection using the semantic segmentation network model of the present application; Figure 17 is a pricked fruit and vegetable image and a recognition image after detection using the semantic segmentation network model of the present application; Figure 18 is a crushed fruit and vegetable image and a recognition image after detection using the semantic segmentation network model of the present application; Figure 19 is a fruit and vegetable grading flowchart of the present application; Figure 20 is a three-fruit and vegetable original image; Figure 21 is a fruit and vegetable contour image extracted using image segmentation technology; Figure 22 is a processing image in which the contour of the fruit and vegetable is fitted using an ellipse fitting algorithm. DETAILED DESCRIPTION

[0022] The preferred embodiment of the fruit and vegetable grading method based on the deep learning model of the present application will be described in detail below with reference to the accompanying drawings: A fruit and vegetable grading method based on a deep learning model, taking gold orange as an example, includes the following steps: (I) Training and learning of neural network model; as shown in Figure 1 , the specific steps include: A1. Batch collection of fruit and vegetable images a; A2. Sample annotation; the specific steps include: A2.1. Preprocessing the fruit and vegetable image a obtained in step A1, the preprocessing operation includes image segmentation processing, binarization processing and noise reduction processing, the image segmentation processing, binarization processing and noise reduction processing are sequentially performed in sequence, the fruit and vegetable part of interest in the image is extracted, and the background part useless for image recognition is removed; A2.2. Cut the image after preprocessing in step A2.1 into small pieces to obtain a plurality of small piece cut images a, divide the cut image a into training set samples, validation set samples and test set samples; A2.3. Perform overall annotation on the cut image a of the training set samples and the validation set samples divided in step A2.2, wherein: The cut image a of the fruitless vegetable defect and no black spot is labeled as the first class, denoted as "good"; The cut image a of the fruit vegetable defect is labeled as the second class, denoted as "bad"; The cut image a of the fruitless vegetable defect but with black spots is labeled as the third class, denoted as "black spot"; The image with more than one color block showing gray is a black spot image, otherwise, the image without any color block showing gray is a non-black spot image; the image with any one of scab, wrinkled skin, stab injury and crush injury is a fruit vegetable defect image, otherwise, the image without any fruit vegetable defect including scab, wrinkled skin, stab injury and crush injury is a non-fruit vegetable defect image; A3. Neural network model training and learning; the specific steps include: A3.1. Select a neural network model; A3.2. Use the cut image a of the training set sample labeled in step A2.3 to train the neural network model determined in step A3.1; Use the cut image a of the validation set sample labeled in step A2.3 to test the neural network model in training, adjust and optimize the parameters of the neural network model; Use the cut image a of the test set sample in step A2.2 to evaluate the trained neural network model; Finally, the deep learning neural network model is obtained; (II) Training and learning of semantic segmentation network model; as shown in Figure 2 The specific steps include: B1. Batch collection of fruit vegetable images b; B2. Sample labeling; the specific steps include: B2.1. Preprocess the fruit vegetable images b collected in step B1, the preprocessing operations include image segmentation processing, binaryzation processing and noise reduction processing, which are performed in sequence, extract the fruit vegetable part of interest in the image, and remove the background part useless for image recognition; the preprocessed fruit vegetable images b are divided into training set samples, test set samples and validation set samples; B2.2. Classify and label the fruit vegetable defects of the fruit vegetable images b in the training set samples and validation set samples preprocessed in step B2.1, and classify the fruit vegetable defects into scab, wrinkled skin, stab injury or crush injury; B3. Semantic segmentation network model training and learning; the specific steps include: B3.1. Select a semantic segmentation network model; B3.2. Using the fruit and vegetable images b from the training set samples labeled in step B2.2, train the semantic segmentation network model determined in step B3.1; Using the fruit and vegetable images b from the validation set samples labeled in step B2.2, the trained semantic segmentation network model is tested, and the parameters of the semantic segmentation network model are adjusted and optimized. Using the fruit and vegetable images b from the test set samples in step B2.1, the trained semantic segmentation network model is evaluated. The final result is a semantic segmentation network model for deep learning; (iii) Grading of fruits and vegetables; such as Figure 3 As shown, the specific steps include: C1. Real-time acquisition of fruit and vegetable images; C2. Preprocess the fruit and vegetable images c acquired in real time in step C1. The preprocessing operations include image segmentation, binarization and noise reduction. Image segmentation, binarization and noise reduction are performed in sequence. Extract the fruit and vegetable parts of interest in the image and remove the background parts that are not useful for image recognition. C3. Cut the preprocessed image from step C2 into small pieces to obtain several small piece images c; C4. Using the deep learning neural network model from step A3.2, identify each slice image c obtained in step C4; C5. Based on the recognition results of step C4, determine whether one or more sliced ​​images c contain fruit and vegetable defects: If none of the sliced ​​images c judged in step C5 have fruit and vegetable defects, then calculate the feature values ​​of the fruits and vegetables in the real-time fruit and vegetable images c acquired in step C1, and classify the fruits and vegetables according to the feature values; the feature values ​​of fruits and vegetables include the area ratio of each color of the fruits and vegetables, fruit shape and volume. If one or more of the segmented images c identified in step C5 contain fruit and vegetable defects, the semantic segmentation network model learned in step B3.2 is used to identify the fruit and vegetable defects in the preprocessed fruit and vegetable images c after step C2 and determine the type of fruit and vegetable defects. After determining the type of fruit and vegetable defects, the fruits and vegetables are first graded according to the type of fruit and vegetable defects. If grading cannot be done by the type of fruit and vegetable defects, the feature values ​​of the fruits and vegetables in the real-time acquired fruit and vegetable images c in step C1 are calculated, and the fruits and vegetables are graded according to the feature values.

[0023] The fruit and vegetable grading method based on the deep learning model needs to train the model before grading. The fruit and vegetable grading method based on the deep learning model uses two models: a neural network model and a semantic segmentation network model, wherein the neural network model is used to identify the fruit and vegetable image and determine whether the fruit and vegetable image has defects; and the semantic segmentation network model is used to identify the fruit and vegetable defects of the fruit and vegetable image and determine the type of the fruit and vegetable defects.

[0024] In the training and learning of the neural network model, the fruit and vegetable image a is first collected in batches through camera shooting, and the collected fruit and vegetable image a is as shown in Figure 4 After that, the sample is labeled, and the neural network model is trained through the sample.

[0025] In the training and learning of the neural network model, in order to obtain complete features of the fruit and vegetable and improve the accuracy of fruit and vegetable grading identification, when the fruit and vegetable image is collected, the fruit and vegetable can be shot from multiple angles through multiple cameras, such as three or more cameras shooting the fruit and vegetable image from the top and both sides. After the fruit and vegetable image is obtained, the obtained fruit and vegetable image can also be subjected to data enhancement processing, including but not limited to image cutting, flipping and random cropping, so as to increase the diversity of the training sample and improve the accuracy of the model training. The data enhancement method includes the methods of cutting, flipping and random cropping.

[0026] In the training and learning of the neural network model, as shown in the obtained fruit and vegetable image a in Figure 4 The fruit and vegetable part to be labeled is the golden orange part in Figure 4 The background part outside the fruit and vegetable part is useless, and in order to avoid the influence of the background part on the labeling work of the fruit and vegetable part. After the fruit and vegetable image a is obtained, the obtained fruit and vegetable image a is first pretreated, and the pretreatment operation includes image segmentation processing. The image after the image segmentation processing is as shown in Figure 5 Only the fruit and vegetable part is retained, and the background part is replaced with black.

[0027] In the training and learning of the neural network model, through segmentation, the fruit and vegetable part of interest in the image is extracted, the background part useless for image recognition is removed, the segmented image is focused on the fruit and vegetable part, background interference is avoided, the high quality of the training sample is ensured, a reliable data basis is provided for model training, the model can be more focused on the extraction of fruit and vegetable features, and the accuracy of the neural network model training and grading identification is improved.

[0028] In the training and learning of the neural network model, the image segmentation processing method in step A2.1 can be but is not limited to a threshold-based segmentation method, a region-based segmentation method, an edge-based segmentation method or a superpixel segmentation method.

[0029] The fruit and vegetable grading method based on the deep learning model, preferably, the image segmentation processing method in step A2.1 is a threshold-based segmentation method. Compared with a region-based segmentation method, an edge-based segmentation method and a superpixel segmentation method, the threshold-based segmentation method only needs to calculate the gray value of a pixel, without complex region or edge analysis, and has simple algorithm, high processing speed and high calculation efficiency, so that the image recognition time can be greatly shortened and the calculation resources can be saved; and the method has low demand for calculation resources, and can run on a resource-limited device (such as an embedded system); in addition, the method mainly depends on threshold selection, involves fewer parameters, and is easy to adjust and optimize.

[0030] In the training and learning of the neural network model, the threshold-based segmentation method can be specifically operated by gray-scale processing of an image and then threshold segmentation processing through a gray histogram. The method divides the pixels in the image into foreground (fruit and vegetable part) and background by setting one or more gray thresholds. The specific implementation steps are as follows: one or more gray thresholds are selected according to the color characteristics (such as the yellow color of gold orange) of the fruit and vegetable image a, the threshold can be manually set, automatically determined based on histogram analysis or an adaptive algorithm (such as Otsu algorithm); compare the gray value of each pixel in the image with the threshold, if the pixel gray value is greater than the threshold, it is classified as foreground (fruit and vegetable part); otherwise, it is classified as background; the pixel value of the background part is set to a fixed value (such as 0 or 255).

[0031] In the training and learning of the neural network model, the threshold-based segmentation method can also be implemented by threshold segmentation of a certain component based on an HSV model. Specifically, the original RGB image is converted into an HSV image, the HSV image can better separate color information and reduce the influence of light changes on the segmentation result; the HSV image includes three components of hue (Hue), saturation (Saturation) and brightness (Value), and the threshold segmentation of the hue (Hue) component H component is taken as an example, the H component (hue) is extracted from the HSV image, the value range of the H component is usually 0° to 360°, which represents the color type (such as red, green, blue, etc.); according to the color characteristics of the fruit and vegetable, the threshold range of the H component is set, for example, for the yellow gold orange, the threshold range of the H component can be set to 20° to 40°; traverse each pixel in the image, judge whether the H component is in the set threshold range, if the H component is in the threshold range, the pixel is classified as foreground (fruit and vegetable part), if the H component is not in the threshold range, the pixel is classified as background; the pixel value of the background part is set to a fixed value (such as 0 or 255).

[0032] In the step of image graying and threshold segmentation processing by a gray histogram, color information is lost when the image is converted into a gray image, only relying on brightness information, and the processing is sensitive to light changes, and brightness changes will directly affect the gray value, resulting in unstable segmentation effect, and it is difficult to process scenes with complex colors or overlapping brightness distribution. When threshold segmentation is performed through the HSV model, the color information is retained, and the segmentation can be performed through the hue (Hue), saturation (Saturation), and brightness (Value) three components, and the HSV model separates the brightness (Value) from the color information (Hue and Saturation), which can reduce the influence of light changes by adjusting the hue and saturation. In addition, different components can be selected for segmentation, which is more flexible and can accurately segment the target through color features.

[0033] In the training and learning of the neural network model, when the HSV model is used for image segmentation processing, the H component among the hue (Hue), saturation (Saturation), and brightness (Value) three components can effectively distinguish different colors of fruits and vegetables, and is particularly suitable for scenes with obvious color characteristics (such as yellow kumquats and red apples). Therefore, in combination with the bright color characteristics of fruit and vegetable images such as kumquat images, preferably, the threshold-based segmentation method is the H component threshold segmentation method based on the HSV model.

[0034] Preferably, the pre-processing operation in step A2.1 further includes binarization processing and noise reduction processing, and the three operations are sequentially performed in the order of image segmentation processing→binarization processing→noise reduction processing to ensure the accuracy and robustness of the segmentation result. Specifically, the segmented image is binarized to convert the image into a black and white binary image, and the binarized image is as shown in Figure 6 The purpose of binarization processing is to further clarify the boundaries between foreground and background, facilitating subsequent noise reduction processing. Then, the binarized image is subjected to noise reduction processing to remove small noise points in the image, and the noise reduction processed image is as shown in Figure 7 The purpose of noise reduction processing is to improve image quality and reduce the interference of noise points on subsequent labeling and model training. After sequentially processing in the order of image segmentation processing→binarization processing→noise reduction processing, the finally obtained pre-processed image is as shown in Figure 8

[0035] ​The fruit and vegetable grading method based on the deep learning model of the present application clearly defines the boundaries of the fruit and vegetable part and the background part through binarization processing, avoiding the influence of fuzzy areas on the segmentation result; through noise reduction processing, small part noise is removed, ensuring the cleanliness and completeness of the segmentation result, and enhancing the image quality. The high-quality image segmentation result after binarization processing and noise reduction processing provides a reliable data basis for subsequent labeling and model training.

[0036] In the noise reduction processing of the fruit and vegetable grading method based on the deep learning model of the present application, the inflation and contraction noise reduction method can be used, or the noise can be reduced by judging the area size. Among them, the inflation and contraction noise reduction method is to inflate and contract the image through morphological operations (such as opening operation and closing operation) to remove small noise points or fill small holes. The noise reduction method based on area size is to analyze the connected regions, count the area of each connected region in the image, and select the target region and noise region according to the set area threshold. Since the inflation and contraction noise reduction method will change the fruit area, and the noise is removed without changing the shape and area of the target region by judging the area size, it is especially suitable for scenes with high accuracy requirements for fruit area, and after combining connected region analysis and area threshold screening, the noise reduction target can be quickly achieved. Therefore, preferably, the noise reduction method based on area size is used in the noise reduction processing.

[0037] In the training and learning of the neural network model of the present application, after preprocessing the fruit and vegetable image a, the image after step A2.1 preprocessing is cut into small pieces to obtain a plurality of small piece cut images a. Then each cut piece image a is labeled as a whole, wherein: the cut piece image a without fruit and vegetable defects and without black spots is labeled as "good"; the cut piece image a with fruit and vegetable defects is labeled as "bad"; and the cut piece image a without fruit and vegetable defects but with black spots is labeled as "black spot". The fruit and vegetable defects are any one of scab, wrinkled skin, stab wound and bruise.

[0038] In the fruit and vegetable grading method based on the deep learning model of the present application, the first implementation of the step of cutting the image after step A2.1 preprocessing into small pieces in step A2.2 is to cut the image after step A2.1 preprocessing in matrix form, and the plurality of cut piece images formed by cutting are as shown in Figure 9 When the image is cut, the area of the block will directly affect the recognition effect. The finer the block, the more detailed information of the cut piece image can be described, and the receptive field of the features such as fruit and vegetable defects or black spots in each cut piece image is larger, but the dimension of feature extraction will increase exponentially, thereby increasing the time complexity of the algorithm. At the same time, too fine block will also make the cut piece image too sparse and lose statistical features, causing overfitting phenomenon and reducing the recognition rate.

[0039] The fruit and vegetable grading method based on the deep learning model is characterized in that the preprocessed images in step A2.1 are cut into small pieces in step A2.2. Figure 10 As shown in the table, the recognition accuracy of the trained neural network model is higher after the images are cut into pieces in the form of a 256*256 matrix. Figure 10 As shown in the table, the recognition accuracy of the trained neural network model is higher after the images are cut into pieces in the form of a 256*256 matrix.

[0040] The second implementation of the step of cutting the preprocessed images in step A2.1 into small pieces in step A2.2 of the fruit and vegetable grading method based on the deep learning model is that the preprocessed images in step A2.1 are divided into pieces in a fan-like manner with the center point of the preprocessed images in step A2.1 as the center of the circle. Figure 11 As shown in the table, the recognition accuracy of the trained neural network model is higher after the images are cut into pieces in the form of a 256*256 matrix. Figure 12 As shown in the table, the recognition accuracy of the trained neural network model is higher after the images are cut into pieces in the form of a 256*256 matrix.

[0041] The step of cutting the preprocessed images in step A2.1 into small pieces in step A2.2 of the fruit and vegetable grading method based on the deep learning model can be cutting in the form of a matrix or dividing into pieces in a fan-like manner.

[0042] When the images are cut in the form of a matrix, each piece of the cut image a has no specific features, some pieces of the cut image a can be all fruit and vegetable parts or all background parts, some pieces of the cut image a can have fruit and vegetable parts and background parts, and the content of the cut image a is chaotic, so that the staff cannot form a more unified and clear cognition of each piece of the cut image a during labeling, especially the black dots on the background parts and the fruit and vegetable parts, which are black and easy to be confused and mislabeled; in addition, if the black dots on the fruit and vegetable parts are on the cutting line, the black dots on the cutting line are usually identified as background parts, so that the black dots on the cutting line cannot be labeled, are missed, and the accuracy of labeling is low.

[0043] When the fan-shaped segmentation is adopted, each segmented cut image a has a fruit and vegetable part and a background part, the fruit and vegetable part is extended outward from the center angle of the two cutting edges of the cut image a and is located in the center angle region of the cut image a, and the background part is located outside the cut image a. Each segmented cut image a has the same and uniform characteristics, which can help the staff form a clearer understanding and make a clear definition of the labeled content, greatly facilitating the labeling and improving the labeling efficiency. In addition, under the feature that the fruit and vegetable part is located in the center angle region of the cut image a and the background part is located outside the cut image a, the region between the fruit and vegetable part and the background part is clearly divided, and whether it is a fruit and vegetable defect with different colors or a black point with the same black color as the background part, they can be clearly distinguished in the cut image a and accurately labeled. Even if there is a black image on the cutting edge of the cut image a, the clear region division between the fruit and vegetable part and the background part can make the black image on the cutting line have a clear region attribution, such as the black image is located on the fruit and vegetable part, which is a black point on the surface of the fruit and vegetable, and the black image is located on the background part, which is the background. This makes the black point on the fruit and vegetable part on the cutting line not be mistaken for the background, which greatly improves the accuracy of labeling.

[0044] In the fruit and vegetable grading method based on the deep learning model, the operation of cutting the image preprocessed in step A2.1 into small pieces in step A2.2 can be realized by using python language and calling OpenCV machine vision library to automatically cut the image. By setting and adjusting the cutting parameters such as the number of rows, the number of columns or the cutting angle under the fan-shaped cutting, the volume and quantity of the cutting can be flexibly controlled. It can not only cut uniformly, but also cut non-uniformly.

[0045] In the training and learning of the neural network model of the application, after the sample labeling work is completed, the neural network model is trained and learned. The neural network model can be a neural network model of ResNet50, VGG16 or AlexNet, which is trained and learned by Figure 10 After comparison, it is not difficult to see that the grading accuracy of the neural network model of ResNet50 is relatively higher. Therefore, preferably, the neural network model of ResNet50 is selected, and after the neural network model of ResNet50 is trained and learned, the neural network model of ResNet50 is used to identify the fruit and vegetable image and determine whether there is a fruit and vegetable defect in the fruit and vegetable image.

[0046] In the training and learning of the neural network model, the cut image a can only be divided into training set samples, and the neural network model is trained through the training set samples; the cut image a can also be divided into training set samples, validation set samples and test set samples, the cut image a of the training set samples labeled in step A2.3 is used to train the neural network model determined in step A3.1; the cut image a of the validation set samples labeled in step A2.3 is used to test the neural network model in training, and the parameters of the neural network model are adjusted and optimized; the cut image a of the test set samples in step A2.2 is used to evaluate the trained neural network model; and finally the neural network model of deep learning is obtained. Compared with only setting the training set samples for training, or setting the training set samples and the validation set samples for training and testing, when the training set samples, the validation set samples and the test set samples are set in the present application, the model has higher accuracy after training through complete and perfect learning such as training, testing and evaluation.

[0047] The neural network model training and learning method of the present application can be manually labeled image by image and pixel by pixel; or as described in the above preferred embodiment, the cut image is labeled after cutting. In the above preferred embodiment of the present application, the fruit and vegetable image a is divided into a plurality of small blocks to form small cut images a, and the boundary of each cut image a is self-defined region. In the training and learning of the neural network model, all contents in the region of the cut image a are taken as the labeling target during labeling, and the cut image a is labeled as a whole. Unlike the traditional method, this labeling method no longer takes the specific features such as fruit and vegetable defects, black spots or fruit stems on the fruit and vegetable image a as the labeling target, but labels each cut image a. In the labeling step, it is not necessary to find and label fruit and vegetable defects, black spots or fruit stems one by one on the image, but only to qualitatively determine and label whether the cut image a has fruit and vegetable defects or black spots. This labeling method avoids the cumbersome process of "quantitative" labeling of fruit and vegetable defects, black spots or fruit stems in the traditional method, significantly simplifies the labeling step, reduces the labeling workload, reduces the labeling work intensity, and significantly improves the labeling efficiency.

[0048] The number of fruit stems on fruits and vegetables is one, the quantity is small, and the image features of the fruit stem are generally uniform, which has less interference on the defect recognition of fruits and vegetables. In contrast, the number of black spots on fruits and vegetables is indefinite, and the quantity is usually large. At the same time, the image features of the black spots are diverse, and the shapes are different, which has greater interference on the defect recognition of fruits and vegetables. In combination with these features of fruits and vegetables, the image labeling method of the fruit and vegetable grading recognition deep learning model training sample provided in the application selects "fruit and vegetable defects" and "black spots" as two core judgment indexes when labeling the cut image a in the case that the fruit and vegetable image a simultaneously exists three features of fruit and vegetable defects, black spots or fruit stems, and divides the labeling result into three categories. On the one hand, the "fruit stem" is not selected as a judgment index because the existence or nonexistence of the fruit stem does not affect the quality of the fruits and vegetables, and thus the absence of the fruit stem will not affect the labeling classification result. This design not only meets the actual needs of fruit and vegetable quality evaluation, but also avoids the complexity caused by labeling the "fruit stem". Compared with the classification method using three judgment indexes of "fruit and vegetable defects", "black spots" and "fruit stem", the application only uses two judgment indexes of "fruit and vegetable defects" and "black spots", which significantly reduces the complexity of labeling, simplifies the process of manual labeling, and improves the labeling efficiency. On the other hand, in the labeling classification, the cut image a without fruit and vegetable defects but with black spots is separately divided into a category, which is specially used for the identification and training of the black spots by the deep learning model. This design enables the model to learn and train the black spots individually, thereby effectively avoiding the misjudgment of the black spots as fruit and vegetable defects by the model, significantly improving the identification accuracy of the black spots by the model, and further improving the overall precision and reliability of the fruit and vegetable grading recognition. Through this optimization, the application guarantees the labeling efficiency while taking into account the accuracy and practicality of the neural network model training.

[0049] Taking a kumquat picture as an example, when the traditional labeling software is used to label the kumquat picture point by point, the time required is different due to the different number of black spots on different kumquat pictures, and it takes about 1-2.5 minutes. It takes about 60 minutes to label 50 kumquat images, and the average labeling time of each kumquat image is 1.2 minutes. When 50 kumquat images are labeled by using the image labeling method of the fruit and vegetable grading recognition deep learning model training sample of the application, 1056 cut images a are obtained by using the 256*256 block method and removing the background part, and it takes 27 minutes to label these 1056 cut images a. The average labeling time of each kumquat image is 0.54 minutes, which is significantly shorter than the average labeling time of 1.2 minutes per block in the traditional method. The labeling time of the labeling method of the application is greatly shortened, and the labeling efficiency is greatly improved.

[0050] The training and learning method of the neural network model of the application only needs image block level labeling, breaks through the inherent mode of the traditional point-by-point pixel extreme labeling (drawing each black point or defect with a brush) of each training image which is extremely time-consuming, realizes "one-time labeling and full-factor coverage", and greatly improves the efficiency and economy of data labeling while ensuring the subsequent model detection accuracy.

[0051] The fruit and vegetable grading method based on the deep learning model of the application, during the training and learning of the semantic segmentation network model, first, the fruit and vegetable images are batch collected by shooting with a camera; then, the fruit and vegetable images are preprocessed, preferably, the preprocessing operation includes image segmentation processing, binaryzation processing and noise reduction processing, the image segmentation processing, binaryzation processing and noise reduction processing are sequentially performed in turn, and then labeled, the fruit and vegetable image after the fruit and vegetable defect type labeling is as shown in Figure 13 The labelme is used as the labeling tool, and the labelme can generate Figure 14 The labeling after the fruit and vegetable image sample is labeled, and then the semantic segmentation network model is trained and learned through the labeled fruit and vegetable image sample, and finally the semantic segmentation network model of the deep learning model is obtained. The fruit and vegetable image collection and preprocessing means are the same as those in the training and learning of the neural network model of the application, and therefore, the application will not be described in detail.

[0052] In the training and learning of the semantic segmentation network model, the semantic segmentation network model can be a DeepLab v3+, PSPNet or UNet semantic segmentation network model, the DeepLab v3+, PSPNet and Unet network are used for model training to determine the surface fruit and vegetable defects of gold orange, and the performances of the several models are as shown in Table 1: Table 1

[0053] As shown in the above table, in the DeepLab v3+, PSPNet and Unet semantic segmentation network models, the evaluation index of the DeepLab v3+ model is the highest, and a relatively good effect is obtained. Therefore, preferably, the semantic segmentation network model is a DeepLab v3+ semantic segmentation network model. The DeepLab v3+ semantic segmentation network model is based on the DeepLab v3 and increases the encoding-decoding module and the Xception main network, the encoding-decoding module can better preserve the detailed information of segmentation and obtain more rich context information, and the Xception main network uses deep convolution to further improve the accuracy and speed of the algorithm.

[0054] In the training and learning of the semantic segmentation network model of the application, the collected fruit and vegetable images b can only be divided into training set samples, and the semantic segmentation network model is trained through the training set samples; the collected fruit and vegetable images b can also be divided into training set samples, validation set samples and test set samples, the fruit and vegetable images b of the training set samples labeled in step B2.2 are used to train the semantic segmentation network model determined in step B3.1; the fruit and vegetable images b of the validation set samples labeled in step B2.2 are used to test the trained semantic segmentation network model, and the parameters of the semantic segmentation network model are adjusted and optimized; the fruit and vegetable images b of the test set samples in step B2.1 are used to evaluate the trained semantic segmentation network model; and finally the semantic segmentation network model of deep learning is obtained. Compared with only setting training set samples for training, or setting training set samples and validation set samples for training and testing, when the training set samples, the validation set samples and the test set samples are set in the application, the model has higher accuracy after training through complete and perfect learning such as training, testing and evaluation.

[0055] In the fruit and vegetable grading method based on the deep learning model of the application, after the fruit and vegetable images c are collected in real time, the real-time collected fruit and vegetable images c are first preprocessed, the preprocessing operations all include binaryzation processing and noise reduction processing, and the image segmentation processing, the binaryzation processing and the noise reduction processing are sequentially performed in sequence; then the images preprocessed in step are cut into small pieces, and finally the neural network model of deep learning in step A3.2 is used to identify the cut piece images c and determine whether there is one or more cut piece images c with fruit and vegetable defects in the cut piece images c. The collection, preprocessing and segmentation into small pieces of the fruit and vegetable images c are the same as the collection, preprocessing and segmentation into small pieces of the fruit and vegetable images a in the training and learning of the neural network model of the application, and the application will not be described in detail.

[0056] The fruit and vegetable grading method based on the deep learning model of the application determines whether there is one or more than one piece of fruit and vegetable defect in the cut image c according to the recognition result. If all the cut images c in each cut image c judged in step C5 have no fruit and vegetable defects, the characteristic value of the fruit and vegetable in the fruit and vegetable image c collected in real time in step C1 is directly calculated, the fruit and vegetable is graded according to the characteristic value of the fruit and vegetable, and the characteristic value of the fruit and vegetable includes the area ratio of each color of the fruit and vegetable, the fruit shape and the volume. According to the international standard, the fruit and vegetable volume is divided into extra-large fruit, large fruit, medium fruit, small fruit and extra-small fruit. For example, N is 10, M is 80, if the area ratio of yellow in each color of the fruit and vegetable is less than 10% and the ratio of the long axis to the short axis of the fruit and vegetable is greater than 80%, and at the same time, the volume of the fruit and vegetable is any one of large fruit or extra-large fruit, then the fruit and vegetable is determined as special grade fruit; if the area ratio of yellow in each color of the fruit and vegetable is less than 10% and the ratio of the long axis to the short axis of the fruit and vegetable is greater than 80%, and at the same time, the volume of the fruit and vegetable is any one of medium fruit, small fruit or extra-small fruit, then the fruit and vegetable is determined as first grade fruit; if the area ratio of yellow in each color of the fruit and vegetable is less than 10% and the ratio of the long axis to the short axis of the fruit and vegetable is greater than 80%, and at the same time, the volume of the fruit and vegetable is medium fruit, then the fruit and vegetable is determined as second grade fruit; if the area ratio of yellow in each color of the fruit and vegetable is less than 10% and the ratio of the long axis to the short axis of the fruit and vegetable is greater than 80%, and at the same time, the volume of the fruit and vegetable is small fruit, then the fruit and vegetable is determined as third grade fruit; if the area ratio of yellow in each color of the fruit and vegetable is less than 10% and the ratio of the long axis to the short axis of the fruit and vegetable is greater than 80%, and at the same time, the volume of the fruit and vegetable is extra-small fruit, then the fruit and vegetable is determined as fourth grade fruit.

[0057] If there is one or more than one piece of fruit and vegetable defect in each cut image c judged in step C5, the fruit and vegetable defects of the fruit and vegetable image c preprocessed in step C2 are recognized by using the semantic segmentation network model of deep learning in step B3.2, the fruit and vegetable defect type is judged, and the fruit and vegetable defect type is divided into scab, wrinkled skin, stab injury and crush injury. Figure 15 The scab fruit and vegetable and the corresponding recognition are as shown in Figure 16 The wrinkled skin fruit and vegetable and the corresponding recognition are as shown in Figure 17 The crush injury fruit and vegetable and the corresponding recognition are as shown in Figure 18As shown; then, the fruits and vegetables are first graded according to the type of defects; when they cannot be graded by the type of defects, the feature values ​​of the fruits and vegetables in the real-time acquired fruit and vegetable image c in step C1 are calculated, and the fruits and vegetables are graded according to the feature values. For example, if the fruit and vegetable defect type includes bruising, the fruit and vegetable is classified as Grade 4; if the fruit and vegetable defect type does not include bruising, but includes either puncture or wrinkling, the fruit and vegetable is classified as Grade 3; if the fruit and vegetable defect type does not include bruising, puncture, or wrinkling, the feature values ​​of the fruit and vegetable in the real-time acquired fruit and vegetable image c in step C1 are calculated, and the fruit and vegetable are graded according to the feature values. In this case, if the fruit and vegetable volume is either small or extra-small, the fruit and vegetable is classified as Grade 3; if the fruit and vegetable volume is either medium, large, or extra-large, but does not meet the requirement that the yellow area ratio of each color is less than 10% and the ratio of the long axis to the short axis of the fruit shape is greater than 80%, the fruit and vegetable is classified as Grade 2; if the fruit and vegetable volume is either medium, large, or extra-large, and the yellow area ratio of each color is less than 10% and the ratio of the long axis to the short axis of the fruit shape is greater than 80%, the fruit and vegetable is classified as Grade 1. The grading process for fruits and vegetables is as follows: Figure 19 As shown.

[0058] This invention presents a fruit and vegetable grading method based on a deep learning model. The steps for calculating the area proportion of each color in fruits and vegetables are as follows: The fruit and vegetable image c, acquired in real-time in step C1, is converted from the RGB color space to the HSV color space; based on the characteristics of the HSV color space, the hue range of each color is defined; the image is thresholded using the defined color range to extract the pixel regions of each color; and the area proportion of each color is calculated. For example, calculating the area proportion of red, orange, and yellow colors can help analyze the maturity or quality of kumquats. Figure 20 Taking the three kumquats shown as an example, the area ratio of each color of the fruit and vegetable is calculated as shown in Table 2 below.

[0059] Table 2

[0060] This invention provides a fruit and vegetable grading method based on a deep learning model. The steps for calculating fruit and vegetable shape are as follows: The fruit and vegetable image c acquired in real-time in step C1 is converted to grayscale; image segmentation techniques are used to extract the contours of the fruit and vegetables, such as... Figure 21 As shown, this image segmentation technique is either thresholding or edge detection; an ellipse fitting algorithm is used to fit the contours of the fruits and vegetables, such as... Figure 22As shown, the parameters of the ellipse are obtained; based on the parameters of the ellipse, the fruit shape of the fruit and vegetables is determined. The ellipse fitting algorithm is as follows: least squares method. When determining the fruit shape based on the ellipse parameters, a fruit shape index can be defined as the ratio of the major axis to the minor axis of the ellipse. Then, based on the value of the fruit shape index, the fruit shape category can be determined: if the fruit shape index > 1.0, it is oblong; 0.9 ≤ fruit shape index < 1.0, it is conical or elliptical; 0.8 ≤ fruit shape index < 0.9, it is round or nearly round; 0.6 ≤ fruit shape index < 0.8, it is oval.

[0061] This invention provides a fruit and vegetable grading method based on a deep learning model. The steps for calculating fruit and vegetable volume include: converting the real-time acquired fruit and vegetable image c (from step C1) to grayscale; and using image segmentation techniques to extract the contours of the fruit and vegetables, such as... Figure 21 The image segmentation technique shown is either thresholding or edge detection; an ellipse fitting algorithm is used to fit the contours of the fruits and vegetables, such as... Figure 22 As shown, the parameters of the ellipse are obtained; based on the parameters of the ellipse, the volume of the fruits and vegetables is calculated.

[0062] This invention presents a fruit and vegetable grading method based on a deep learning model. This method employs a segmented grading strategy to improve the accuracy and efficiency of grading. Traditional fruit and vegetable grading methods typically identify defects and characteristic values ​​in a single step and then grade the fruit or vegetable accordingly. In contrast, this invention uses a phased processing approach to progressively determine the quality grade of the fruit or vegetable, thereby significantly reducing computational load while maintaining grading accuracy.

[0063] Specifically, the grading method of the present invention includes the following three stages: Phase 1: Preliminary Fruit and Vegetable Defect Detection First, a deep learning neural network model is used to identify the sliced ​​images c of fruits and vegetables, determining whether at least one sliced ​​image c contains defects. If all sliced ​​images c are free of defects, the process proceeds to feature value calculation. After calculating the feature values ​​of the fruit and vegetable images, the fruits and vegetables are graded based on these feature values. The advantage of this stage is that it reduces the amount of subsequent computation by quickly screening for fruits and vegetables without defects.

[0064] Phase Two: Identification and Grading of Fruit and Vegetable Defect Types If at least one sliced ​​image c is found to contain fruit and vegetable defects, a deep learning semantic segmentation network model is used to accurately identify the defects in image c and determine their specific types. The fruits and vegetables are then graded based on their defect types. The core of this stage is to accurately identify the types of fruit and vegetable defects through semantic segmentation technology and use this as a crucial basis for grading, thereby improving the precision of the grading process.

[0065] Phase 3: Eigenvalue-assisted hierarchical classification In the case where the fruits and vegetables cannot be graded only by the fruit and vegetable defect type, the feature value of the fruit and vegetable image c is further calculated, and the fruits and vegetables are graded according to the feature value. This stage is supplemented to ensure that accurate grading can still be achieved in complex cases.

[0066] The segmented grading strategy described above, on the one hand, in addition to the feature value of the fruits and vegetables, also uses the semantic segmentation network model of deep learning to identify the fruit and vegetable defect type of the fruit and vegetable image c, introduces the fruit and vegetable defect type as the grading basis, significantly increases the diversity of judgment factors, and makes the grading result more accurate and accurate. On the other hand, through the segmented selection strategy, the invention can use the lightweight deep learning neural network model with “qualitative” properties to identify and judge the presence or absence of fruit and vegetable defects at an early stage, quickly screen fruits and vegetables without fruit and vegetable defects, and let special fruits pass quickly. Release computing resources; only non-special fruits call “heavy firepower” type semantic segmentation network model with “quantitative” properties for fine analysis and identification of fruit and vegetable defect types, and grade using fruit and vegetable defect types. This design not only avoids unnecessary feature value calculation, greatly reduces the amount of calculation, improves the grading efficiency, but also enables more intelligent use of resources under a fixed average computing budget, realizes adaptive allocation of computing resources (neural network model and semantic segmentation network model), ensures the smoothness of the overall pipeline, and avoids the occurrence of lag caused by a few complex fruits and vegetables.

[0067] Taking the classification of gold orange as an example, under the traditional one-stage (one-time) classification method, the fruit and vegetable defects and feature values of each gold orange image need to be judged and calculated, and the time required for each gold orange image is not much different. The average time required for each gold orange image to be graded after inputting the model is about 0.65 seconds. When using the neural network model ResNet50, the cutting block method is a matrix form of 256*256, and the semantic segmentation network model is DeepLab v3+, the fruit and vegetable grading method based on the deep learning model of the invention takes about 9.08 seconds to grade 20 gold orange image samples. The grading time of the 20 gold orange image samples is shown in Table Three.

[0068] Table Three

[0069] As can be seen from the above Table Three, the 20 gold orange image samples took a total of 9.08 seconds to grade, and the average time required for each gold orange image sample to be graded was about 0.454 seconds. Compared with the average grading time of 0.65 seconds for each gold orange image in the traditional method, the grading time of the segmented grading method of the invention is greatly shortened, and the grading efficiency is greatly improved.

[0070] In addition, 150 samples of each type of gold orange were detected, and the test results are shown in Table Four.

[0071] Table Four

[0072] From the above table four, when the fruit and vegetable quality is graded by using the grading method of the application, the average accuracy is 98.53%, and the grading effect is good.

[0073] The fruit and vegetable grading method based on the deep learning model simplifies the image labeling work to the image block level labeling by using the block strategy, significantly reduces the workload level of image labeling, and reduces the labeling cost. On the other hand, in the grading strategy aspect, the segmented grading strategy is used to realize the adaptive allocation of computing resources (neural network model and semantic segmentation network model), which reduces the computing workload and makes the whole grading calculation more smooth. The fruit and vegetable grading method based on the deep learning model reduces the overall grading workload in a magnitude order from image labeling to grading strategy, and greatly improves the work efficiency.

[0074] In addition, in the fruit and vegetable grading method based on the deep learning model, the fruit and vegetable image is blocked when the neural network model training sample is labeled, and the whole is labeled. The neural network model only needs to focus on learning the relatively simple task of "whether the cut image exists defects", which is easier to train and more robust than making a semantic segmentation network model learn positioning and classification of all dense small targets at the same time, and the accuracy of the neural network model training is also higher, so that under the same sample set and computing budget, the grading accuracy of the fruit and vegetable image is also relatively high.

[0075] The application is simple for ordinary skilled person in the technical field of the application without departing from the concept of the application, and some simple deductions or substitutions can be made, which should be regarded as belonging to the protection scope of the application.

Claims

1. A fruit and vegetable grading method based on a deep learning model, characterized in that: Includes the following steps: C1. Real-time acquisition of fruit and vegetable images; C2. Preprocess the fruit and vegetable images c acquired in real time in step C1. The preprocessing operation includes image segmentation: extracting the fruit and vegetable parts of interest in the image and removing the background parts that are not useful for image recognition. C3. Cut the preprocessed image from step C2 into small pieces to obtain several small piece images c; C4. Using a neural network model, identify each slice image c obtained in step C4; C5. Based on the recognition results of step C4, determine whether one or more sliced ​​images c contain fruit and vegetable defects: If none of the sliced ​​images c determined in step C5 have fruit and vegetable defects, then the feature values ​​of the fruits and vegetables in the real-time fruit and vegetable images c acquired in step C1 are calculated, and the fruits and vegetables are graded according to the feature values. The feature values ​​of fruits and vegetables include one or more of the following: the area ratio of each color of the fruit and vegetable, fruit shape, and volume. If one or more of the sliced ​​images c identified in step C5 contain fruit and vegetable defects, then the semantic segmentation network model is used to identify the fruit and vegetable defects in the preprocessed fruit and vegetable images c after step C2 and determine the type of fruit and vegetable defects. After determining the type of fruit and vegetable defects, the fruits and vegetables are first graded according to the type of fruit and vegetable defects. If the fruit and vegetable defects cannot be graded by type, then the feature values ​​of the fruits and vegetables in the real-time acquired fruit and vegetable images c in step C1 are calculated, and the fruits and vegetables are graded according to the feature values.

2. The fruit and vegetable grading method based on a deep learning model according to claim 1, characterized in that: In step C2, the preprocessing operations all include binarization and noise reduction. Image segmentation, binarization, and noise reduction are performed sequentially.

3. The fruit and vegetable grading method based on a deep learning model according to claim 1, characterized in that: The neural network model in step C4 is a deep learning neural network model obtained after training and learning; The steps involved in training and learning a neural network model include: A1. Batch acquisition of fruit and vegetable images a; A2. Sample labeling; specific steps include: A2.

1. Preprocess the fruit and vegetable image a obtained in step A1. The preprocessing operation includes image segmentation: extracting the fruit and vegetable parts of interest in the image and removing the background parts that are not useful for image recognition; A2.

2. Cut the image after preprocessing in step A2.1 into small pieces to obtain several small piece images a, and divide the piece images a into training set samples; A2.

3. Perform overall annotation on the sliced ​​images a of the training set samples divided in step A2.2, where: Image 'a', which contains no fruit or vegetable defects and no black spots, is labeled as Class I. Image 'a' of a cut piece of fruit or vegetable with defects is labeled as category 2. Image 'a' of a diced fruit or vegetable without defects but with black spots is labeled as category 3. An image with one or more gray patches is considered an image with black spots; conversely, an image without any gray patches is considered an image without black spots. An image with one or more of the following fruit and vegetable defects is considered an image with fruit and vegetable defects; conversely, an image without any of these defects is considered an image without fruit and vegetable defects. A3. Neural network model training and learning; specific steps include: A3.

1. Select a neural network model; A3.

2. Using the sliced ​​image a of the training set samples labeled in step A2.3, train the neural network model determined in step A3.1; finally, obtain the deep learning neural network model.

4. The fruit and vegetable grading method based on a deep learning model according to claim 3, characterized in that: In the training and learning of the neural network model, the step of cutting the image after preprocessing in step A2.1 into small pieces in step A2.2 is to use the center point of the image as the center of the circle and perform fan-shaped segmentation on the image after preprocessing in step A2.

1.

5. The fruit and vegetable grading method based on a deep learning model according to claim 1, characterized in that: The semantic segmentation network model in step C5 is a deep learning semantic segmentation network model obtained after training and learning. The steps involved in training and learning a semantic segmentation network model include: B1. Batch acquisition of fruit and vegetable images; B2. Sample labeling; specific steps include: B2.

1. Preprocess the fruit and vegetable images b collected in step B1; divide the preprocessed fruit and vegetable images b into training set samples; B2.

2. Classify and label the fruit and vegetable defects in the fruit and vegetable images b in the training set samples after preprocessing in step B2.1, and classify the fruit and vegetable defects into scabs, wrinkles, punctures or bruises; B3. Semantic segmentation network model training and learning; specific steps include: B3.

1. Select a semantic segmentation network model; B3.

2. Using the fruit and vegetable images b from the training set samples labeled in step B2.2, train the semantic segmentation network model determined in step B3.1; finally, obtain the deep learning semantic segmentation network model.

6. The fruit and vegetable grading method based on a deep learning model according to claim 1, characterized in that: In step C5, the feature values ​​of fruits and vegetables include the area proportion of each color. The steps to calculate the area proportion of each color are as follows: convert the fruit and vegetable image c acquired in real time in step C1 from the RGB color space to the HSV color space; define the hue range of each color according to the characteristics of the HSV color space; use the defined color range to perform threshold segmentation on the image and extract the pixel region of each color respectively; calculate the area proportion of each color.

7. The fruit and vegetable grading method based on a deep learning model according to claim 1, characterized in that: In step C5, the feature values ​​of fruits and vegetables include their shape. The steps for calculating the shape of fruits and vegetables are as follows: the fruit and vegetable image c acquired in real time in step C1 is converted to grayscale; the contour of the fruits and vegetables is extracted using image segmentation technology, which can be either threshold segmentation or edge detection; the contour of the fruits and vegetables is fitted using an ellipse fitting algorithm to obtain the parameters of the ellipse; and the shape of the fruits and vegetables is determined based on the parameters of the ellipse.

8. The fruit and vegetable grading method based on a deep learning model according to claim 1, characterized in that: In step C5, the feature values ​​of fruits and vegetables include their volume. The steps for calculating the volume of fruits and vegetables are as follows: the fruit and vegetable image c acquired in real time in step C1 is converted to grayscale; the contour of the fruits and vegetables is extracted using image segmentation technology, which can be either threshold segmentation or edge detection; the contour of the fruits and vegetables is fitted using an ellipse fitting algorithm to obtain the parameters of the ellipse; and the volume of the fruits and vegetables is calculated based on the parameters of the ellipse.

9. The fruit and vegetable grading method based on a deep learning model according to claim 1, characterized in that: In step C5, the characteristic values ​​of fruits and vegetables include the area ratio of each color, fruit shape, and volume; the types of fruit and vegetable defects are divided into scabs, wrinkles, punctures, and bruises.

10. The fruit and vegetable grading method based on a deep learning model according to claim 1, characterized in that: Step C5 involves determining, based on the recognition results of step C4, whether one or more sliced ​​images c contain fruit or vegetable defects. If none of the sliced ​​images c determined in step C5 have fruit or vegetable defects, then the feature values ​​of the fruits and vegetables in the real-time acquired fruit and vegetable images c in step C1 are calculated, and the fruits and vegetables are graded according to their feature values: if the proportion of yellow area in each color of the fruit or vegetable is less than N%, and the ratio of the long axis to the short axis of the fruit or vegetable shape is greater than M%, and the fruit or vegetable volume is either large or extra-large, then the fruit or vegetable is classified as premium grade; if the proportion of yellow area in each color of the fruit or vegetable is less than N%, and the ratio of the long axis to the short axis of the fruit or vegetable shape is greater than M%, and the fruit or vegetable volume is either medium, small, or extra-small. If the yellow area is less than N% and the ratio of the long axis to the short axis of the fruit shape is greater than M%, and the fruit is of medium size, then the fruit is classified as Grade 1 fruit. If the yellow area is less than N% and the ratio of the long axis to the short axis of the fruit shape is greater than M%, and the fruit is of small size, then the fruit is classified as Grade 3 fruit. If the yellow area is less than N% and the ratio of the long axis to the short axis of the fruit shape is greater than M%, and the fruit is of extra-small size, then the fruit is classified as Grade 4 fruit. If one or more of the segmented images c identified in step C5 contain fruit and vegetable defects, then the semantic segmentation network model learned in step B3.2 is used to identify the fruit and vegetable defects in the real-time acquired fruit and vegetable images c in step C1 and determine the type of defect. After determining the type of defect, the fruit and vegetable are first graded according to the defect type: if the defect type includes bruising, the fruit and vegetable are classified as grade four; if the defect type does not include bruising, but includes either puncture or wrinkling, the fruit and vegetable are classified as grade three; if the defect type does not include bruising, puncture, or wrinkling, the fruit and vegetable are classified as grade three. Then, calculate the feature values ​​of the fruits and vegetables in the real-time acquired fruit and vegetable image c in step C1, and classify the fruits and vegetables according to the feature values. At this time, if the fruit and vegetable volume is either small or extra small, the fruit and vegetable are judged as grade 3 fruits; if the fruit and vegetable volume is either medium, large or extra large, but does not meet the requirements that the proportion of yellow area in each color of the fruit and vegetable is less than N% and the ratio of the long axis to the short axis of the fruit and vegetable shape is greater than M%, the fruit and vegetable are judged as grade 2 fruits; if the fruit and vegetable volume is either medium, large or extra large, and the proportion of yellow area in each color of the fruit and vegetable is less than N% and the ratio of the long axis to the short axis of the fruit and vegetable shape is greater than M%, the fruit and vegetable are judged as grade 1 fruits. N and M are natural numbers greater than 0 and less than 100.

Images