A fruit and vegetable grading method based on a deep learning model

Abstract

This invention discloses a fruit and vegetable grading method based on a deep learning model, which progressively determines the quality grade of fruits and vegetables through staged processing. This staged grading strategy, on the one hand, utilizes a deep learning semantic segmentation network model to identify the types of fruit and vegetable defects in the images, in addition to the feature values ​​of the fruits and vegetables. Introducing these defect types as a grading criterion significantly increases the diversity of judgment factors, resulting in more refined and accurate grading results. On the other hand, through staged processing, this invention can quickly screen for fruits and vegetables without defects in the early stages and prioritize grading based on the types of defects, avoiding unnecessary feature value calculations, thereby greatly reducing computational load and improving grading efficiency.

Description

Technical Field

[0001] This invention relates to the field of automatic fruit and vegetable grading technology, and in particular to a fruit and vegetable grading method based on a deep learning model. Background Technology

[0002] In recent years, many scholars at home and abroad have studied machine vision-based methods for inspecting the quality of agricultural products. However, traditional machine learning methods for grading the surface quality of fruits and vegetables have drawbacks, such as the need for detailed human knowledge, interference from human judgment factors, long computation time, and low accuracy when processing large amounts of data.

[0003] With the development of science and technology, deep learning data models have gradually emerged and been applied to image recognition. The complex network structure and data training models not only reduce the workload but also improve the speed and accuracy of detection. Fruit and vegetable grading based on deep learning models mainly involves two aspects: model training and the identification and grading of fruits and vegetables.

[0004] In model training, the annotation of training samples is a key task that directly determines the accuracy of the trained model. In fruit and vegetable grading, the annotation work for existing models is primarily done manually, which is extremely time-consuming and labor-intensive. This is especially true for some fruits and vegetables, such as kumquats and pears, which, in addition to defects like scabs and bruises, also have stems and numerous black spots. Both stems and black spots are easily misidentified as defects during annotation. Furthermore, to achieve high recognition accuracy, the training model requires a large number of black spot samples. Therefore, the annotation process must differentiate and annotate each spot individually, significantly increasing the workload and difficulty. This makes the annotation work very demanding, eye-straining, and inefficient, resulting in low accuracy and negatively impacting the model training results.

[0005] In the identification and grading of fruits and vegetables, the current conventional approach is to use models to detect and identify or calculate the quality characteristics of fruits and vegetables, such as volume, shape, color, and surface defects, and then grade them all at once based on these characteristic values. Under this grading method, every quality characteristic of the fruit and vegetable, except for surface defects, such as volume, shape, and color, needs to be identified or calculated. The computational workload is enormous, especially when there are many quality characteristics, where the computational workload increases exponentially. This huge computational burden severely impacts grading efficiency.

[0006] In summary, existing fruit and vegetable grading methods based on deep learning models have heavy workloads in both model training and identification and grading, resulting in low grading efficiency, and need further improvement. Summary of the Invention

[0007] The purpose of this invention is to provide a fruit and vegetable grading method based on a deep learning model.

[0008] The technical solution for achieving the objective of this invention is: a fruit and vegetable grading method based on a deep learning model, characterized by comprising the following steps:

[0009] C1. Real-time acquisition of fruit and vegetable images;

[0010] C2. Preprocess the fruit and vegetable images c acquired in real time in step C1. The preprocessing operation includes image segmentation: extracting the fruit and vegetable parts of interest in the image and removing the background parts that are not useful for image recognition.

[0011] C3. Cut the preprocessed image from step C2 into small pieces to obtain several small piece images c;

[0012] C4. Using a neural network model, identify each slice image c obtained in step C4;

[0013] C5. Based on the recognition results of step C4, determine whether one or more sliced ​​images c contain fruit and vegetable defects:

[0014] If none of the sliced ​​images c determined in step C5 have fruit and vegetable defects, then the feature values ​​of the fruits and vegetables in the real-time fruit and vegetable images c acquired in step C1 are calculated, and the fruits and vegetables are graded according to the feature values. The feature values ​​of fruits and vegetables include one or more of the following: the area ratio of each color of the fruit and vegetable, fruit shape, and volume.

[0015] If one or more of the sliced ​​images c identified in step C5 contain fruit and vegetable defects, then the semantic segmentation network model is used to identify the fruit and vegetable defects in the preprocessed fruit and vegetable images c after step C2 and determine the type of fruit and vegetable defects. After determining the type of fruit and vegetable defects, the fruits and vegetables are first graded according to the type of fruit and vegetable defects. If the fruit and vegetable defects cannot be graded by type, then the feature values ​​of the fruits and vegetables in the real-time acquired fruit and vegetable images c in step C1 are calculated, and the fruits and vegetables are graded according to the feature values.

[0016] Furthermore, the neural network model in step C4 is a deep learning neural network model obtained after training and learning; the steps of training and learning the neural network model include:

[0017] A1. Batch acquisition of fruit and vegetable images a;

[0018] A2. Sample labeling; specific steps include:

[0019] A2.1. Preprocess the fruit and vegetable image a obtained in step A1. The preprocessing operation includes image segmentation: extracting the fruit and vegetable parts of interest in the image and removing the background parts that are not useful for image recognition;

[0020] A2.2. Cut the image after preprocessing in step A2.1 into small pieces to obtain several small piece images a, and divide the piece images a into training set samples;

[0021] A2.3. Perform overall annotation on the sliced ​​images a of the training set samples divided in step A2.2, where:

[0022] Image 'a', which contains no fruit or vegetable defects and no black spots, is labeled as Class I.

[0023] Image 'a' of a cut piece of fruit or vegetable with defects is labeled as category 2.

[0024] Image 'a' of a diced fruit or vegetable without defects but with black spots is labeled as category 3.

[0025] An image with one or more gray patches is considered an image with black spots; conversely, an image without any gray patches is considered an image without black spots. An image with one or more of the following fruit and vegetable defects is considered an image with fruit and vegetable defects; conversely, an image without any of these defects is considered an image without fruit and vegetable defects.

[0026] A3. Neural network model training and learning; specific steps include:

[0027] A3.1. Select a neural network model;

[0028] A3.2. Using the sliced ​​image a of the training set samples labeled in step A2.3, train the neural network model determined in step A3.1; finally, obtain the deep learning neural network model.

[0029] Furthermore, in the training and learning of the neural network model, the step of cutting the image after preprocessing in step A2.1 into small pieces in step A2.2 is to perform fan-shaped segmentation on the image after preprocessing in step A2.1 with the center point of the image as the center.

[0030] Further, in step A2.2, the sliced ​​image a is divided into training set samples, validation set samples, and test set samples; in step A2.3, the sliced ​​images a of the training set samples and validation set samples in step A2.2 are labeled as a whole; step A2.3 is to train the neural network model determined in step A3.1 using the labeled sliced ​​images a of the training set samples in step A2.3; step A3.2 is to test the trained neural network model and adjust and optimize the parameters of the neural network model using the labeled sliced ​​images a of the validation set samples in step A2.3; and to evaluate the trained neural network model using the sliced ​​images a of the test set samples in step A2.2; finally, the deep learning neural network model is obtained.

[0031] Furthermore, the semantic segmentation network model in step C5 is a deep learning semantic segmentation network model obtained after training and learning; the steps for training and learning the semantic segmentation network model include:

[0032] B1. Batch acquisition of fruit and vegetable images;

[0033] B2. Sample labeling; specific steps include:

[0034] B2.1. Preprocess the fruit and vegetable images b collected in step B1; divide the preprocessed fruit and vegetable images b into training set samples;

[0035] B2.2. Classify and label the fruit and vegetable defects in the fruit and vegetable images b in the training set samples after preprocessing in step B2.1, and classify the fruit and vegetable defects into scabs, wrinkles, punctures or bruises;

[0036] B3. Semantic segmentation network model training and learning; specific steps include:

[0037] B3.1. Select a semantic segmentation network model;

[0038] B3.2. Using the fruit and vegetable images b from the training set samples labeled in step B2.2, train the semantic segmentation network model determined in step B3.1; finally, obtain the deep learning semantic segmentation network model.

[0039] Further, in step B2.1, the preprocessed fruit and vegetable image b is divided into training set samples, validation set samples, and test set samples; step B2.2 involves classifying and labeling the fruit and vegetable defects in the preprocessed training set samples and validation set samples of the fruit and vegetable image b; step B3.2 involves training the semantic segmentation network model determined in step B3.1 using the fruit and vegetable images b of the training set samples labeled in step B2.2; testing the trained semantic segmentation network model using the fruit and vegetable images b of the validation set samples labeled in step B2.2, and adjusting and optimizing the parameters of the semantic segmentation network model; evaluating the trained semantic segmentation network model using the fruit and vegetable images b of the test set samples in step B2.1; finally, the semantic segmentation network model of the deep learning model is obtained.

[0040] Furthermore, in step B2.1, the preprocessing operation includes image segmentation, extracting the fruit and vegetable parts of interest in the image, and removing the background parts that are not useful for image recognition; in steps A2.1, B2.1 and C2, the preprocessing operation also includes binarization and noise reduction, and image segmentation, binarization and noise reduction are performed in sequence.

[0041] Further, in step C5, the feature values ​​of fruits and vegetables include the area proportion of each color. The steps for calculating the area proportion of each color are as follows: convert the fruit and vegetable image c acquired in real time in step C1 from the RGB color space to the HSV color space; define the hue range of each color according to the characteristics of the HSV color space; use the defined color range to perform threshold segmentation on the image, and extract the pixel region of each color respectively; calculate the area proportion of each color. For example, calculate the area proportion values ​​of red, orange, and yellow colors to analyze the maturity or quality of kumquats.

[0042] Further, in step C5, the feature values ​​of fruits and vegetables include their shape. The steps for calculating the fruit and vegetable shape are as follows: The fruit and vegetable image c acquired in real-time in step C1 is converted to grayscale; the contour of the fruit and vegetable is extracted using image segmentation technology, which can be either threshold segmentation or edge detection; the contour of the fruit and vegetable is fitted using an ellipse fitting algorithm to obtain the parameters of the ellipse; and the fruit and vegetable shape is determined based on the ellipse parameters. The ellipse fitting algorithm can be the least squares method. When determining the fruit and vegetable shape based on the ellipse parameters, a fruit shape index can be defined as the ratio of the major axis to the minor axis of the ellipse. Then, based on the value of the fruit shape index, the fruit and vegetable shape category can be determined: if the fruit shape index > 1.0, it is oblong; 0.9 ≤ fruit shape index < 1.0, it is conical or elliptical; 0.8 ≤ fruit shape index < 0.9, it is round or nearly round; 0.6 ≤ fruit shape index < 0.8, it is oval.

[0043] Further, in step C5, the feature values ​​of fruits and vegetables include their volume. The steps for calculating the volume of fruits and vegetables are as follows: The fruit and vegetable image c acquired in real-time in step C1 is converted to grayscale; the contours of the fruits and vegetables are extracted using image segmentation techniques, which can be either thresholding or edge detection; the contours of the fruits and vegetables are fitted using an ellipse fitting algorithm to obtain the parameters of the ellipse; and the volume of the fruits and vegetables is calculated based on the ellipse parameters. According to international standards, fruits and vegetables are classified into extra-large, large, medium, small, and extra-small fruits.

[0044] Furthermore, in step C5, the characteristic values ​​of fruits and vegetables include the area ratio of each color, fruit shape, and volume; the types of fruit and vegetable defects are divided into scabs, wrinkles, punctures, and pressure marks.

[0045] Further, step C5 involves determining, based on the recognition result of step C4, whether one or more sliced ​​images c contain fruit or vegetable defects:

[0046] If none of the sliced ​​images c determined in step C5 have fruit or vegetable defects, then the feature values ​​of the fruits and vegetables in the real-time acquired fruit and vegetable images c in step C1 are calculated, and the fruits and vegetables are graded according to their feature values: if the proportion of yellow area in each color of the fruit or vegetable is less than N%, and the ratio of the long axis to the short axis of the fruit or vegetable shape is greater than M%, and the fruit or vegetable volume is either large or extra-large, then the fruit or vegetable is classified as premium grade; if the proportion of yellow area in each color of the fruit or vegetable is less than N%, and the ratio of the long axis to the short axis of the fruit or vegetable shape is greater than M%, and the fruit or vegetable volume is either medium, small, or extra-small. If the yellow area is less than N% and the ratio of the long axis to the short axis of the fruit shape is greater than M%, and the fruit is of medium size, then the fruit is classified as Grade 1 fruit. If the yellow area is less than N% and the ratio of the long axis to the short axis of the fruit shape is greater than M%, and the fruit is of small size, then the fruit is classified as Grade 3 fruit. If the yellow area is less than N% and the ratio of the long axis to the short axis of the fruit shape is greater than M%, and the fruit is of extra-small size, then the fruit is classified as Grade 4 fruit.

[0047] If one or more of the segmented images c identified in step C5 contain fruit and vegetable defects, the semantic segmentation network model learned in step B3.2 is used to identify the fruit and vegetable defects in the real-time acquired fruit and vegetable images c in step C1 and determine the type of defect. After determining the type of defect, the fruits and vegetables are first graded according to the defect type: if the defect type includes bruising, the fruit and vegetable are classified as grade four; if the defect type does not include bruising, but includes either puncture or wrinkling, the fruit and vegetable are classified as grade three; if the defect type does not include bruising, puncture, or wrinkling, then... Calculate the feature values ​​of fruits and vegetables in the real-time acquired fruit and vegetable image c in step C1, and classify the fruits and vegetables according to the feature values. At this time, if the fruit and vegetable volume is either small or extra small, the fruit and vegetable are judged as grade 3 fruits; if the fruit and vegetable volume is either medium, large or extra large, but does not meet the requirements that the proportion of yellow area in each color of the fruit and vegetable is less than N% and the ratio of the long axis to the short axis of the fruit and vegetable shape is greater than M%, the fruit and vegetable are judged as grade 2 fruits; if the fruit and vegetable volume is either medium, large or extra large, and the proportion of yellow area in each color of the fruit and vegetable is less than N% and the ratio of the long axis to the short axis of the fruit and vegetable shape is greater than M%, the fruit and vegetable are judged as grade 1 fruits.

[0048] N and M are natural numbers greater than 0 and less than 100.

[0049] This invention presents a fruit and vegetable grading method based on a deep learning model. Compared to traditional methods that identify fruit and vegetable defects and feature values ​​in a single step and then grade them accordingly, this method uses a phased processing approach to progressively determine the quality grade of fruits and vegetables. This phased grading strategy, on the one hand, utilizes a deep learning semantic segmentation network model to identify the types of fruit and vegetable defects in the images, introducing these defect types as a grading criterion. This significantly increases the diversity of judgment factors, resulting in more refined and accurate grading results. On the other hand, through phased processing, this invention can quickly screen for defect-free fruits and vegetables in the early stages and prioritize grading based on defect types, avoiding unnecessary feature value calculations, thereby greatly reducing computational load and improving grading efficiency. Attached Figure Description

[0050] Figure 1 This is a flowchart of the training and learning process of the neural network model of this invention;

[0051] Figure 2 This is a flowchart of the training and learning process of the semantic segmentation network model of this invention;

[0052] Figure 3 This is a flowchart of the fruit and vegetable grading method based on a deep learning model according to the present invention;

[0053] Figure 4This is the original image of fruit and vegetable image a used in the training and learning of the neural network model of this invention;

[0054] Figure 5 yes Figure 4 The image obtained after image segmentation processing;

[0055] Figure 6 yes Figure 5 The image obtained after binarization;

[0056] Figure 7 yes Figure 6 The image obtained after noise reduction processing;

[0057] Figure 8 yes Figure 4 The final image obtained after preprocessing;

[0058] Figure 9 The image a is a block image obtained by dividing the preprocessed image into blocks in matrix form during the training and learning of the neural network model of this invention.

[0059] Figure 10 This is a comparison chart of the recognition accuracy of three models, ResNet50, VGG16, and AlexNet, under different numbers of matrix blocks during the training and learning of the neural network model of this invention.

[0060] Figure 11 This is a schematic diagram of fan-shaped segmentation of a preprocessed image during the training and learning of the neural network model of this invention;

[0061] Figure 12 The image a is a single-block image obtained by performing fan-shaped segmentation on the preprocessed image during the training and learning of the neural network model of this invention.

[0062] Figure 13 This is fruit and vegetable image a after the defect types of fruits and vegetables are labeled in the fruit and vegetable grading method based on deep learning model of this invention;

[0063] Figure 14 This is the labeled image generated from the labeled fruit and vegetable image a in the fruit and vegetable grading method based on a deep learning model of this invention;

[0064] Figure 15 These are images of scab-covered fruits and vegetables and their corresponding recognition images detected using the semantic segmentation network model of this invention;

[0065] Figure 16 These are images of wrinkled fruits and vegetables and their corresponding recognition images detected using the semantic segmentation network model of this invention.

[0066] Figure 17 These are images of punctured fruits and vegetables and their corresponding recognition images after detection using the semantic segmentation network model of this invention.

[0067] Figure 18 These are images of bruised fruits and vegetables and their corresponding recognition images after detection using the semantic segmentation network model of this invention.

[0068] Figure 19 This is a flowchart of the fruit and vegetable grading process of the present invention;

[0069] Figure 20 These are the original images of three fruits and vegetables;

[0070] Figure 21 It uses image segmentation technology to extract the outline of fruits and vegetables;

[0071] Figure 22 This is a processed image of fruits and vegetables whose contours are fitted using an ellipse fitting algorithm. Detailed Implementation

[0072] The preferred embodiment of the fruit and vegetable grading method based on a deep learning model of the present invention will be described in detail below with reference to the accompanying drawings:

[0073] A fruit and vegetable grading method based on a deep learning model, taking kumquats as an example, includes the following steps:

[0074] (a) Training and learning of neural network models; such as Figure 1 As shown, the specific steps include:

[0075] A1. Batch acquisition of fruit and vegetable images a;

[0076] A2. Sample labeling; specific steps include:

[0077] A2.1. Preprocess the fruit and vegetable image a obtained in step A1. The preprocessing operation includes image segmentation, binarization and noise reduction. The image segmentation, binarization and noise reduction are performed in sequence. Extract the fruit and vegetable parts of interest in the image and remove the background parts that are not useful for image recognition.

[0078] A2.2. Cut the image after preprocessing in step A2.1 into small pieces to obtain several small piece images a. Divide the piece images a into training set samples, validation set samples and test set samples.

[0079] A2.3. Perform overall annotation on the sliced ​​images 'a' of the training set samples and validation set samples divided in step A2.2, where:

[0080] Image 'a' of cut vegetables without defects or black spots is labeled as Class 1 and denoted as "good";

[0081] Image 'a' of a cut piece of fruit or vegetable with defects is labeled as Category 2 and marked as "bad";

[0082] Image 'a' of a cut piece of fruit or vegetable without defects but with black spots is labeled as category 3 and denoted as "black spot".

[0083] An image with one or more gray patches is considered an image with black spots; conversely, an image without any gray patches is considered an image without black spots. An image with any of the following fruit and vegetable defects—scabs, wrinkles, punctures, and bruises—is considered an image with fruit and vegetable defects; conversely, an image without any of these defects is considered an image without fruit and vegetable defects.

[0084] A3. Neural network model training and learning; specific steps include:

[0085] A3.1. Select a neural network model;

[0086] A3.2. Using the sliced ​​image a of the training set samples labeled in step A2.3, train the neural network model determined in step A3.1;

[0087] Using the sliced ​​image a of the validation set samples labeled in step A2.3, the neural network model under training is tested, and the parameters of the neural network model are adjusted and optimized.

[0088] The trained neural network model is evaluated using the sliced ​​image a of the test set samples in step A2.2.

[0089] The final result is a deep learning neural network model;

[0090] (ii) Training and learning of semantic segmentation network models; such as Figure 2 As shown, the specific steps include:

[0091] B1. Batch acquisition of fruit and vegetable images;

[0092] B2. Sample labeling; specific steps include:

[0093] B2.1. Preprocess the fruit and vegetable image b acquired in step B1. The preprocessing operations include image segmentation, binarization, and noise reduction. Image segmentation, binarization, and noise reduction are performed sequentially. Extract the fruit and vegetable parts of interest from the image and remove the background parts that are not useful for image recognition. The preprocessed fruit and vegetable image b is divided into training set samples, test set samples, and validation set samples.

[0094] B2.2. Classify and label the fruit and vegetable defects in the preprocessed training set samples and validation set samples in step B2.1, and classify the fruit and vegetable defects into scabs, wrinkles, punctures or bruises.

[0095] B3. Semantic segmentation network model training and learning; specific steps include:

[0096] B3.1. Select a semantic segmentation network model;

[0097] B3.2. Using the fruit and vegetable images b from the training set samples labeled in step B2.2, train the semantic segmentation network model determined in step B3.1;

[0098] Using the fruit and vegetable images b from the validation set samples labeled in step B2.2, the trained semantic segmentation network model is tested, and the parameters of the semantic segmentation network model are adjusted and optimized.

[0099] Using the fruit and vegetable images b from the test set samples in step B2.1, the trained semantic segmentation network model is evaluated.

[0100] The final result is a semantic segmentation network model for deep learning;

[0101] (iii) Grading of fruits and vegetables; such as Figure 3 As shown, the specific steps include:

[0102] C1. Real-time acquisition of fruit and vegetable images;

[0103] C2. Preprocess the fruit and vegetable images c acquired in real time in step C1. The preprocessing operations include image segmentation, binarization and noise reduction. Image segmentation, binarization and noise reduction are performed in sequence. Extract the fruit and vegetable parts of interest in the image and remove the background parts that are not useful for image recognition.

[0104] C3. Cut the preprocessed image from step C2 into small pieces to obtain several small piece images c;

[0105] C4. Using the deep learning neural network model from step A3.2, identify each slice image c obtained in step C4;

[0106] C5. Based on the recognition results of step C4, determine whether one or more sliced ​​images c contain fruit and vegetable defects:

[0107] If none of the sliced ​​images c judged in step C5 have fruit and vegetable defects, then calculate the feature values ​​of the fruits and vegetables in the real-time fruit and vegetable images c acquired in step C1, and classify the fruits and vegetables according to the feature values; the feature values ​​of fruits and vegetables include the area ratio of each color of the fruits and vegetables, fruit shape and volume.

[0108] If one or more of the segmented images c identified in step C5 contain fruit and vegetable defects, the semantic segmentation network model learned in step B3.2 is used to identify the fruit and vegetable defects in the preprocessed fruit and vegetable images c after step C2 and determine the type of fruit and vegetable defects. After determining the type of fruit and vegetable defects, the fruits and vegetables are first graded according to the type of fruit and vegetable defects. If grading cannot be done by the type of fruit and vegetable defects, the feature values ​​of the fruits and vegetables in the real-time acquired fruit and vegetable images c in step C1 are calculated, and the fruits and vegetables are graded according to the feature values.

[0109] This invention presents a fruit and vegetable grading method based on a deep learning model, which requires training the model before grading. The invention utilizes two models: a neural network model and a semantic segmentation network model. The neural network model is used to identify fruit and vegetable images and determine the presence of defects; the semantic segmentation network model is used to identify fruit and vegetable defects and determine their types.

[0110] This invention relates to a fruit and vegetable grading method based on a deep learning model. During the training and learning of the neural network model, fruit and vegetable images 'a' are first acquired in batches via a camera. The acquired fruit and vegetable images 'a' are shown below. Figure 4 As shown; then, labeled samples are used to train the neural network model.

[0111] In the training and learning of the neural network model of this invention, to obtain complete features of fruits and vegetables and improve the accuracy of fruit and vegetable grading recognition, multiple cameras can be used to capture images of fruits and vegetables from multiple angles when acquiring images, such as using three or more cameras to acquire images of fruits and vegetables from the top and sides. After acquiring the fruit and vegetable images, data augmentation processing can be performed on the acquired fruit and vegetable images, including but not limited to image cropping, flipping, and random cropping, to increase the diversity of training samples and improve the accuracy of model training. Among them, the data augmentation methods include cropping, flipping, and random cropping.

[0112] In the training and learning of the neural network model of this invention, such as Figure 4 In the obtained fruit and vegetable image a shown, the parts to be labeled are the fruit and vegetable portions, i.e. Figure 4 The yellow kumquat portion is highlighted, while the background outside the fruit and vegetable portion is irrelevant and should not be included in the labeling process. After acquiring the fruit and vegetable image 'a', it undergoes preprocessing, including image segmentation. The segmented image is shown below. Figure 5 As shown, only the fruit and vegetable portion is retained, while the background portion is replaced with black.

[0113] In the training and learning of the neural network model of this invention, the fruit and vegetable parts of interest in the image are extracted through segmentation, and the background parts that are not useful for image recognition are removed, so that the segmented image focuses on the fruit and vegetable parts, avoids background interference, ensures the high quality of training samples, provides a reliable data foundation for model training, and the model can focus more on the extraction of fruit and vegetable features, thereby improving the accuracy of hierarchical recognition in neural network model training.

[0114] In the training and learning of the neural network model of the present invention, the image segmentation processing method in step A2.1 may be, but is not limited to, threshold-based segmentation, region-based segmentation, edge-based segmentation, or superpixel segmentation.

[0115] This invention provides a fruit and vegetable grading method based on a deep learning model. Preferably, the image segmentation method in step A2.1 is a threshold-based segmentation method. Compared with region-based segmentation, edge-based segmentation, and superpixel segmentation, the threshold-based segmentation method only needs to calculate the grayscale value of pixels, without requiring complex region or edge analysis. The algorithm is simple, fast, and computationally efficient, significantly reducing image recognition time and saving computational resources. Furthermore, this method has lower computational resource requirements and can run on resource-constrained devices (such as embedded systems). In addition, it mainly relies on threshold selection, involves fewer parameters, and is easy to adjust and optimize.

[0116] In the training and learning of the neural network model of this invention, the threshold-based segmentation method can be implemented by converting the image to grayscale and then performing threshold segmentation using a grayscale histogram. This method divides the pixels in the image into foreground (fruit and vegetable parts) and background by setting one or more grayscale thresholds. The specific implementation steps are as follows: Based on the color characteristics of the fruit and vegetable image 'a' (e.g., the yellow of a kumquat), select one or more grayscale thresholds. The thresholds can be manually set, determined automatically based on histogram analysis, or by an adaptive algorithm (e.g., the Otsu algorithm); compare the grayscale value of each pixel in the image with the threshold. If the pixel's grayscale value is greater than the threshold, it is classified as foreground (fruit and vegetable parts); otherwise, it is classified as background; set the pixel values ​​of the background part to fixed values ​​(e.g., 0 or 255).

[0117] In the training and learning of the neural network model of this invention, the threshold-based segmentation method can also be implemented based on the HSV model to achieve threshold segmentation of a certain component. Specifically, the original RGB image is converted into an HSV image. HSV images can better separate color information and reduce the impact of lighting changes on the segmentation results. HSV images include three components: hue, saturation, and value. Taking threshold segmentation of the hue component (H component) as an example, the H component (hue) is extracted from the HSV image. The value range of the H component is usually from 0° to 360°, representing the type of color (such as red, green, blue, etc.). According to the color characteristics of fruits and vegetables, the threshold range of the H component is set. For example, for yellow kumquats, the threshold range of the H component can be set to 20° to 40°. Each pixel in the image is traversed to determine whether its H component is within the set threshold range. If the H component is within the threshold range, the pixel is classified as foreground (fruit and vegetable part). If the H component is not within the threshold range, the pixel is classified as background. The pixel values ​​of the background part are set to fixed values ​​(such as 0 or 255).

[0118] In the process of converting an image to grayscale and then using a grayscale histogram for thresholding, color information is lost during the conversion, relying only on brightness information. This makes the process sensitive to changes in lighting conditions, as brightness variations directly affect grayscale values, leading to unstable segmentation results and difficulty in handling scenes with complex colors or overlapping brightness distributions. In contrast, thresholding using the HSV model preserves color information. It can segment using three components: hue, saturation, and value. The HSV model separates value from color information (hue and saturation), allowing it to reduce the impact of lighting changes by adjusting hue and saturation. Furthermore, it offers greater flexibility by selecting different components for segmentation, enabling accurate target segmentation based on color features.

[0119] In the training and learning of the neural network model of this invention, when using the HSV model for image segmentation, among the three components of hue, saturation, and value, the H component can effectively distinguish fruits and vegetables of different colors, and is particularly suitable for scenes with obvious color features (such as yellow kumquats, red apples, etc.). Therefore, considering the vivid color features of fruit and vegetable images such as kumquat images, a threshold-based segmentation method is preferably the H component threshold segmentation method based on the HSV model.

[0120] The present invention provides a fruit and vegetable grading method based on a deep learning model. Preferably, the preprocessing operation in step A2.1 further includes binarization and noise reduction. These three operations are performed sequentially in the order of image segmentation → binarization → noise reduction to ensure the accuracy and robustness of the segmentation results. Specifically, the segmented image is binarized to convert it into a black and white binary image, as shown in the image below. Figure 6 As shown, the purpose of binarization is to further clarify the boundary between the foreground and background, facilitating subsequent noise reduction processing. Then, noise reduction is performed on the binarized image to remove small portions of noise. The noise-reduced image is shown below. Figure 7 As shown, the purpose of noise reduction is to improve image quality and reduce the interference of noise on subsequent annotation and model training. After processing in the order of image segmentation → binarization → noise reduction, the final preprocessed image is shown below. Figure 8 As shown.

[0121] This invention presents a fruit and vegetable grading method based on a deep learning model. Binarization clearly defines the boundary between the fruit / vegetable portion and the background, avoiding the influence of blurred areas on the segmentation result. Noise reduction removes minor noise, ensuring a clean and complete segmentation result and enhancing image quality. The high-quality image segmentation results after binarization and noise reduction provide a reliable data foundation for subsequent annotation and model training.

[0122] This invention presents a fruit and vegetable grading method based on a deep learning model. During noise reduction, it can employ either dilation / contraction noise reduction or noise reduction based on area size. The dilation / contraction noise reduction method uses morphological operations (such as opening and closing operations) to dilate and shrink the image, removing small noise points or filling small holes. The area-based noise reduction method uses connected component analysis to statistically analyze the area of ​​each connected region in the image and filters out target and noise regions based on a set area threshold. Since dilation / contraction noise reduction alters the fruit area, noise reduction based on area size can remove noise without changing the shape and area of ​​the target region. This is particularly suitable for scenarios requiring high accuracy in fruit area measurement, and by combining connected component analysis and area threshold filtering, the noise reduction target can be achieved quickly. Therefore, preferably, the area-based noise reduction method is used during noise reduction.

[0123] In the training and learning of the neural network model of this invention, after preprocessing the fruit and vegetable image a, the image preprocessed in step A2.1 is cut into small pieces to obtain several small piece images a. Then, each piece image a is labeled as follows: a piece image a without fruit and vegetable defects and without black spots is labeled "good"; a piece image a with fruit and vegetable defects is labeled "bad"; and a piece image a without fruit and vegetable defects but with black spots is labeled "black spots". Fruit and vegetable defects can be any one of the following: scabs, wrinkles, punctures, and pressure marks.

[0124] The present invention provides a fruit and vegetable grading method based on a deep learning model. In step A2.2, the first implementation of the step of slicing the preprocessed image from step A2.1 into small blocks involves slicing the preprocessed image from step A2.1 into blocks in a matrix format. The resulting block images are shown below. Figure 9 As shown, when slicing an image, the area of ​​the slice directly affects the recognition effect. The finer the slice, the more detailed the information of the sliced ​​image can be described, and the larger the receptive field of features such as fruit and vegetable defects or black spots in each sliced ​​image is. However, the dimension of feature extraction will increase exponentially, thereby increasing the time complexity of the algorithm. At the same time, overly fine slices will also make the sliced ​​image too sparse and lose statistical features, causing overfitting and reducing the recognition rate.

[0125] This invention presents a fruit and vegetable grading method based on a deep learning model. The fruit and vegetable slices are divided into blocks using matrices of 64*64, 128*128, and 256*256 respectively. The labeled images after slicing are then used to train three neural network models: ResNet50, VGG16, and AlexNet. The recognition accuracy of the three neural network models after training is shown below. Figure 10 As shown. By Figure 10 It is easy to see that the trained neural network model achieves higher recognition accuracy after being divided into blocks in a 256*256 matrix format. Therefore, in the fruit and vegetable grading method based on a deep learning model of this invention, preferably, in step A2.2, the first implementation of the step of dividing the preprocessed image from step A2.1 into small blocks is to divide the preprocessed image from step A2.1 into blocks in a 256*256 matrix format. With a 256*256 block image, both the amount of annotation and the accuracy of the annotation are considered. While ensuring that the amount of annotation is not too large, the receptive field of targets such as fruit and vegetable defects or black spots in the block image is still relatively large, and the annotation accuracy is still relatively high.

[0126] This invention relates to a fruit and vegetable grading method based on a deep learning model. A second implementation of step A2.2, which involves segmenting the image preprocessed in step A2.1 into smaller pieces, involves using the center point of the image preprocessed in step A2.1 as the center of a circle, and performing fan-shaped segmentation on the image, such as... Figure 11As shown. Each segmented image 'a' includes two cutting edges. The intersection of these two cutting edges is the center point of the preprocessed image A2.1. The center point of the preprocessed image A2.1 falls on the fruit and vegetable portion. The segmented single image 'a' is shown below. Figure 12 As shown.

[0127] The present invention provides a fruit and vegetable grading method based on a deep learning model. In step A2.2, the step of cutting the image after preprocessing in step A2.1 into small blocks can be performed by cutting the blocks in a matrix form or by performing fan-shaped segmentation.

[0128] When segmenting in a matrix format, each segmented image 'a' lacks specific features. Some image 'a' may consist entirely of fruits and vegetables or entirely of background, while others may contain both. The content of image 'a' is disorganized, making it difficult for staff to form a unified and clear understanding of each image 'a' during annotation. In particular, the black dots on the background and fruit and vegetable parts are all black, making them easy to confuse and mislabel. Furthermore, if a black dot on the fruit and vegetable part happens to appear on the cutting line, it is usually identified as background because it is located at the edge of image 'a', resulting in the black dots on the cutting line not being labeled, leading to omissions and low annotation accuracy.

[0129] When using a fan-shaped segmentation method, each segmented image 'a' has a fruit / vegetable portion and a background portion. The fruit / vegetable portion extends outward from the central corner of the two cutting edges of image 'a' and is located in the central corner region of image 'a'. The background portion is located outside image 'a'. Each segmented image 'a' has the same and uniform characteristics. These characteristics allow staff to form a clearer understanding and make clear definitions of the content to be labeled, greatly facilitating labeling and improving labeling efficiency. In addition, with the characteristic that "the fruit / vegetable portion is located in the central corner region of image 'a' and the background portion is located outside image 'a'", the region between the fruit / vegetable portion and the background portion is clearly defined. Whether it is a defect in fruit / vegetable with different colors or a black dot that is the same black as the background portion, it can be clearly distinguished in image 'a' and accurately labeled. Even if a black image appears on the cutting edge of the slice image a, the clear division between the fruit and vegetable part and the background part can give the black image on the cutting line a clear regional classification. If the black image is on the fruit and vegetable part, it is a black dot on the surface of the fruit and vegetable. If the black image is on the background part, it is the background. This prevents the black dots of the fruit and vegetable part that appear on the cutting line from being mistaken for the background, which greatly improves the accuracy of the annotation.

[0130] This invention presents a fruit and vegetable grading method based on a deep learning model. Step A2.2, which involves slicing the preprocessed image from step A2.1 into small pieces, can be performed using Python and the OpenCV machine vision library to achieve automatic image slicing. By setting and adjusting slicing parameters such as the number of rows, columns, or the cutting angle under a fan-shaped slicing pattern, the volume and number of slices can be flexibly controlled. It can perform both uniform and non-uniform slicing.

[0131] In the training and learning of the neural network model of this invention, after the sample labeling is completed, the neural network model is trained and learned. The neural network model can be a ResNet50, VGG16, or AlexNet neural network model, and is... Figure 10 After comparison, it is easy to see that the ResNet50 neural network model has a relatively higher classification accuracy. Therefore, it is preferable to select the ResNet50 neural network model, train and learn the ResNet50 neural network model, and then use the ResNet50 neural network model to identify fruit and vegetable images and determine whether there are defects in the fruit and vegetable images.

[0132] In the training and learning of the neural network model of this invention, during training and learning, the sliced ​​image 'a' can be divided into only training set samples, and the neural network model can be trained using these training set samples. Alternatively, sliced ​​image 'a' can be divided into training set samples, validation set samples, and test set samples. Using sliced ​​image 'a' of the training set samples labeled in step A2.3, the neural network model determined in step A3.1 is trained. Using sliced ​​image 'a' of the validation set samples labeled in step A2.3, the trained neural network model is tested, and the parameters of the neural network model are adjusted and optimized. Using sliced ​​image 'a' of the test set samples in step A2.2, the trained neural network model is evaluated. Finally, a deep learning neural network model is obtained. Compared to training with only training set samples, or training and testing with both training and validation set samples, this invention, by setting training set samples, validation set samples, and test set samples, achieves a higher accuracy rate after a complete and comprehensive learning process including training, testing, and evaluation.

[0133] The training and learning method of the neural network model of this invention can be carried out manually, image by image, pixel by pixel; or, as described in the preferred embodiment above, by segmenting the image and then annotating the segmented images as a whole. In the preferred embodiment above, the fruit and vegetable image a is divided into multiple small blocks to form segmented images a, and the boundaries of each segmented image a are self-defined regions. During the training and learning of the neural network model, during annotation, all content within the region of segmented image a is used as the annotation target, and the segmented image a is holistically labeled. Unlike traditional methods, this annotation method no longer uses specific features such as fruit and vegetable defects, black spots, or stems on the fruit and vegetable image a as annotation targets, but instead annotates each segmented image a unit. In the annotation step, it is not necessary to search for and annotate fruit and vegetable defects, black spots, or stems one by one on the image; it is only necessary to qualitatively determine and annotate whether fruit and vegetable defects or black spots exist in the segmented image a. This labeling method avoids the tedious process of "quantitatively" labeling defects, black spots, or fruit stems on fruits and vegetables one by one in the traditional method. It significantly simplifies the labeling steps, reduces the workload, lowers the intensity of labeling work, and greatly improves labeling efficiency.

[0134] Fruit stems are typically few in number and their image features are generally uniform, thus causing minimal interference with defect identification. In contrast, black spots on fruits and vegetables are numerous and varied in image features and shape, significantly interfering with defect identification. Based on these characteristics, the image annotation method for training samples of the deep learning model for fruit and vegetable grading identification proposed in this invention selects "fruit and vegetable defects" and "black spots" as two core judgment indicators when annotating the sliced ​​image a, where fruit and vegetable image a simultaneously possesses three features: defects, black spots, or stems. The annotation results are then categorized into three types. This labeling method, on the one hand, does not use "fruit stems" as a criterion, because the presence or absence of fruit stems does not affect the quality of fruits and vegetables, and therefore their absence will not affect the labeling and classification results. This design not only meets the actual needs of fruit and vegetable quality evaluation, but also avoids the added complexity caused by labeling "fruit stems." Compared with classification methods that use three criteria—"fruit and vegetable defects," "black spots," and "fruit stems"—this invention uses only two criteria: "fruit and vegetable defects" and "black spots," significantly reducing the complexity of labeling, simplifying the manual labeling process, and improving labeling efficiency. On the other hand, in the labeling and classification, this invention classifies the sliced ​​image 'a' with black spots but no fruit and vegetable defects into a separate category, specifically for training the deep learning model to identify black spots. This design allows the model to learn and train separately for black spots, effectively avoiding the model misclassifying black spots as fruit and vegetable defects, significantly improving the model's accuracy in identifying black spots in fruits and vegetables, and further improving the overall accuracy and reliability of fruit and vegetable grading and identification. Through this optimization, this invention ensures labeling efficiency while also taking into account the accuracy and practicality of neural network model training.

[0135] Taking kumquat images as an example, traditional annotation software is used to annotate kumquat images point by point. Since the number of black dots varies across different kumquat images, the time required is approximately 1-2.5 minutes. Annotating 50 kumquat images would take about 60 minutes, with an average annotation time of 1.2 minutes per image. However, when using the image annotation method of the fruit and vegetable grading recognition deep learning model training samples of this invention to annotate 50 kumquat images, the images are divided into 256*256 blocks. After removing the background, 1056 sliced ​​images (a) are obtained. Annotating these 1056 sliced ​​images (a) takes 27 minutes. The average time spent annotating each of the 50 kumquat images is 0.54 minutes. Compared to the traditional average annotation time of 1.2 minutes per block, the annotation time of this invention is significantly reduced, and the annotation efficiency is greatly improved.

[0136] The training and learning method of the neural network model of this invention only requires image patch-level annotation, breaking through the inherent mode of traditional point-by-point pixel-level annotation of each training image (drawing each black dot or defect bit by bit with a pen), realizing "one-time annotation, full element coverage", which greatly improves the efficiency and economy of data annotation while ensuring the detection accuracy of subsequent models.

[0137] This invention relates to a fruit and vegetable grading method based on a deep learning model. During the training and learning of the semantic segmentation network model, fruit and vegetable images are first acquired in batches via a camera. Preprocessing of these images is then performed, preferably including image segmentation, binarization, and noise reduction. These processes are performed sequentially, followed by labeling. The resulting fruit and vegetable images, labeled with defect types, are shown below. Figure 13 As shown, the annotation tool is LabelMe. LabelMe's built-in tools can generate annotations from fruit and vegetable images. Figure 14 The annotations are shown below; then, the semantic segmentation network model is trained and learned using the annotated fruit and vegetable image samples, and finally, the semantic segmentation network model of the deep learning model is obtained. The methods for acquiring and preprocessing the fruit and vegetable images are the same as those used in the training and learning of the neural network model of this invention, and will not be elaborated further in this invention.

[0138] In the training and learning of the semantic segmentation network model of this invention, the semantic segmentation network model can be a DeepLab v3+, PSPNet, or UNet semantic segmentation network model. DeepLab v3+, PSPNet, and UNet networks were used for model training to determine defects on the surface of kumquats. The performance of the several models is shown in Table 1 below:

[0139] Table 1

[0140]

[0141] As shown in the table above, among the semantic segmentation network models DeepLab v3+, PSPNet, and Unet, DeepLab v3+ achieves the highest evaluation metrics, demonstrating superior performance. Therefore, this invention preferably selects the DeepLab v3+ semantic segmentation network model. The DeepLab v3+ semantic segmentation network model adds an encoder-decoder module and an Xception backbone network to the DeepLab v3 model. The addition of the encoder-decoder module allows for better preservation of segmentation details and the acquisition of richer contextual information; the addition of the Xception backbone network, employing deep convolution, further improves the algorithm's accuracy and speed.

[0142] In the training and learning of the semantic segmentation network model of this invention, the collected fruit and vegetable image b can be divided into training set samples, and the semantic segmentation network model can be trained using these training set samples. Alternatively, the collected fruit and vegetable image b can be divided into training set samples, validation set samples, and test set samples. The fruit and vegetable image b of the training set samples labeled in step B2.2 is used to train the semantic segmentation network model determined in step B3.1. The fruit and vegetable image b of the validation set samples labeled in step B2.2 is used to test the trained semantic segmentation network model, and the parameters of the semantic segmentation network model are adjusted and optimized. The fruit and vegetable image b of the test set samples in step B2.1 is used to evaluate the trained semantic segmentation network model. Finally, a deep learning semantic segmentation network model is obtained. Compared with training only using training set samples, or training and testing using both training and validation set samples, this invention, by setting training set samples, validation set samples, and test set samples, achieves a higher accuracy rate after a complete and comprehensive learning process including training, testing, and evaluation.

[0143] This invention relates to a fruit and vegetable grading method based on a deep learning model. During fruit and vegetable grading, after real-time acquisition of fruit and vegetable images (c), the acquired images (c) are first preprocessed. The preprocessing operations include binarization and noise reduction. Image segmentation, binarization, and noise reduction are performed sequentially. The preprocessed images are then segmented into small pieces. Finally, the deep learning neural network model from step A3.2 is used to identify each segmented image (c) and determine whether one or more segments (c) contain fruit or vegetable defects. The methods for acquiring, preprocessing, and segmenting fruit and vegetable images (c) are the same as those used for acquiring, preprocessing, and segmenting fruit and vegetable images (a) during the training and learning of the neural network model in this invention. Therefore, this invention will not elaborate further on these methods.

[0144] This invention relates to a fruit and vegetable grading method based on a deep learning model. According to the recognition results, it determines whether one or more sliced ​​images c contain fruit and vegetable defects. If, in step C5, all sliced ​​images c have no fruit and vegetable defects, then the feature values ​​of the fruits and vegetables in the real-time acquired fruit and vegetable images c in step C1 are directly calculated, and the fruits and vegetables are graded according to their feature values. The feature values ​​of fruits and vegetables include the area ratio of each color, fruit shape, and volume. According to international standards, the volume of fruits and vegetables is divided into extra-large fruit, large fruit, medium fruit, small fruit, and extra-small fruit. For example, if N is 10 and M is 80, and the yellow area accounts for less than 10% of the total area of ​​the fruit or vegetable, and the ratio of the long axis to the short axis of the fruit or vegetable is greater than 80%, and the fruit or vegetable is either large or extra-large, then the fruit or vegetable is classified as premium grade. If the yellow area accounts for less than 10% of the total area of ​​the fruit or vegetable, and the ratio of the long axis to the short axis of the fruit or vegetable is greater than 80%, and the fruit or vegetable is either medium, small, or extra-small, then the fruit or vegetable is classified as grade 1. If the yellow area does not meet the requirements... If the yellow area is less than 10% and the ratio of the long axis to the short axis of the fruit shape is greater than 80%, and the fruit volume is medium, then the fruit is classified as Grade 2. If the yellow area is less than 10% and the ratio of the long axis to the short axis of the fruit shape is greater than 80%, and the fruit volume is small, then the fruit is classified as Grade 3. If the yellow area is less than 10% and the ratio of the long axis to the short axis of the fruit shape is greater than 80%, and the fruit volume is extra small, then the fruit is classified as Grade 4.

[0145] If one or more of the segmented images c identified in step C5 contain fruit and vegetable defects, then the semantic segmentation network model from deep learning in step B3.2 is used to identify the fruit and vegetable defects in the preprocessed images c from step C2, determining the type of defect. Fruit and vegetable defect types are categorized into scabs, wrinkles, puncture wounds, and pressure bruises. The identification of scab-covered fruits and vegetables and their corresponding defects is as follows: Figure 15 As shown, wrinkled fruits and vegetables and their corresponding identification methods are as follows: Figure 16 As shown, puncture wounds on fruits and vegetables and their corresponding identification methods are as follows: Figure 17 As shown, bruised fruits and vegetables and their corresponding identification are as follows: Figure 18As shown; then, the fruits and vegetables are first graded according to the type of defects; when they cannot be graded by the type of defects, the feature values ​​of the fruits and vegetables in the real-time acquired fruit and vegetable image c in step C1 are calculated, and the fruits and vegetables are graded according to the feature values. For example, if the fruit and vegetable defect type includes bruising, the fruit and vegetable is classified as Grade 4; if the fruit and vegetable defect type does not include bruising, but includes either puncture or wrinkling, the fruit and vegetable is classified as Grade 3; if the fruit and vegetable defect type does not include bruising, puncture, or wrinkling, the feature values ​​of the fruit and vegetable in the real-time acquired fruit and vegetable image c in step C1 are calculated, and the fruit and vegetable are graded according to the feature values. In this case, if the fruit and vegetable volume is either small or extra-small, the fruit and vegetable is classified as Grade 3; if the fruit and vegetable volume is either medium, large, or extra-large, but does not meet the requirement that the yellow area ratio of each color is less than 10% and the ratio of the long axis to the short axis of the fruit shape is greater than 80%, the fruit and vegetable is classified as Grade 2; if the fruit and vegetable volume is either medium, large, or extra-large, and the yellow area ratio of each color is less than 10% and the ratio of the long axis to the short axis of the fruit shape is greater than 80%, the fruit and vegetable is classified as Grade 1. The grading process for fruits and vegetables is as follows: Figure 19 As shown.

[0146] This invention presents a fruit and vegetable grading method based on a deep learning model. The steps for calculating the area proportion of each color in fruits and vegetables are as follows: The fruit and vegetable image c, acquired in real-time in step C1, is converted from the RGB color space to the HSV color space; based on the characteristics of the HSV color space, the hue range of each color is defined; the image is thresholded using the defined color range to extract the pixel regions of each color; and the area proportion of each color is calculated. For example, calculating the area proportion of red, orange, and yellow colors can help analyze the maturity or quality of kumquats. Figure 20 Taking the three kumquats shown as an example, the area ratio of each color of the fruit and vegetable is calculated as shown in Table 2 below.

[0147] Table 2

[0148]

[0149] This invention provides a fruit and vegetable grading method based on a deep learning model. The steps for calculating fruit and vegetable shape are as follows: The fruit and vegetable image c acquired in real-time in step C1 is converted to grayscale; image segmentation techniques are used to extract the contours of the fruit and vegetables, such as... Figure 21 As shown, this image segmentation technique is either thresholding or edge detection; an ellipse fitting algorithm is used to fit the contours of the fruits and vegetables, such as... Figure 22As shown, the parameters of the ellipse are obtained; based on the parameters of the ellipse, the fruit shape of the fruit and vegetables is determined. The ellipse fitting algorithm is as follows: least squares method. When determining the fruit shape based on the ellipse parameters, a fruit shape index can be defined as the ratio of the major axis to the minor axis of the ellipse. Then, based on the value of the fruit shape index, the fruit shape category can be determined: if the fruit shape index > 1.0, it is oblong; 0.9 ≤ fruit shape index < 1.0, it is conical or elliptical; 0.8 ≤ fruit shape index < 0.9, it is round or nearly round; 0.6 ≤ fruit shape index < 0.8, it is oval.

[0150] This invention provides a fruit and vegetable grading method based on a deep learning model. The steps for calculating fruit and vegetable volume include: converting the real-time acquired fruit and vegetable image c (from step C1) to grayscale; and using image segmentation techniques to extract the contours of the fruit and vegetables, such as... Figure 21 The image segmentation technique shown is either thresholding or edge detection; an ellipse fitting algorithm is used to fit the contours of the fruits and vegetables, such as... Figure 22 As shown, the parameters of the ellipse are obtained; based on the parameters of the ellipse, the volume of the fruits and vegetables is calculated.

[0151] This invention presents a fruit and vegetable grading method based on a deep learning model. This method employs a segmented grading strategy to improve the accuracy and efficiency of grading. Traditional fruit and vegetable grading methods typically identify defects and characteristic values ​​in a single step and then grade the fruit or vegetable accordingly. In contrast, this invention uses a phased processing approach to progressively determine the quality grade of the fruit or vegetable, thereby significantly reducing computational load while maintaining grading accuracy.

[0152] Specifically, the grading method of the present invention includes the following three stages:

[0153] Phase 1: Preliminary Fruit and Vegetable Defect Detection

[0154] First, a deep learning neural network model is used to identify the sliced ​​images c of fruits and vegetables, determining whether at least one sliced ​​image c contains defects. If all sliced ​​images c are free of defects, the process proceeds to feature value calculation. After calculating the feature values ​​of the fruit and vegetable images, the fruits and vegetables are graded based on these feature values. The advantage of this stage is that it reduces the amount of subsequent computation by quickly screening for fruits and vegetables without defects.

[0155] Phase Two: Identification and Grading of Fruit and Vegetable Defect Types

[0156] If at least one sliced ​​image c is found to contain fruit and vegetable defects, a deep learning semantic segmentation network model is used to accurately identify the defects in image c and determine their specific types. The fruits and vegetables are then graded based on their defect types. The core of this stage is to accurately identify the types of fruit and vegetable defects through semantic segmentation technology and use this as a crucial basis for grading, thereby improving the precision of the grading process.

[0157] Phase 3: Eigenvalue-assisted hierarchical classification

[0158] When it is not possible to classify fruits and vegetables solely based on defect types, feature values ​​of the fruit and vegetable image c are further calculated, and the fruits and vegetables are classified according to these feature values. This stage serves as a supplement to ensure accurate classification even in complex situations.

[0159] The aforementioned segmented grading strategy, on the one hand, in addition to the feature values ​​of fruits and vegetables, also utilizes a deep learning semantic segmentation network model to identify the types of fruit and vegetable defects in image c, introducing these defect types as a grading criterion. This significantly increases the diversity of judgment factors, making the grading results more refined and accurate. On the other hand, through the segmented sorting strategy, this invention can use a lightweight, qualitative deep learning neural network model to identify the presence or absence of fruit and vegetable defects in the early stages, quickly filtering out defect-free fruits and vegetables, allowing premium-grade fruits to pass quickly and freeing up computing resources. Only for non-premium-grade fruits are processed using a quantitative, high-powered semantic segmentation network model to perform detailed analysis and identification of the types of fruit and vegetable defects, and then grading them based on these defect types. This design not only avoids unnecessary feature value calculations, significantly reducing computational load and improving grading efficiency, but also enables more intelligent resource utilization under a fixed average computational budget, achieving adaptive allocation of computing resources (neural network model and semantic segmentation network model), ensuring the smoothness of the overall pipeline, and avoiding bottlenecks caused by a few complex fruits and vegetables.

[0160] Taking kumquat grading as an example, under the traditional one-time grading method, the defects and feature values ​​of each kumquat image need to be judged and calculated. The time required for each kumquat image is not much different, and the average time for grading calculation after inputting each kumquat image into the model is about 0.65 seconds. However, the fruit and vegetable grading method based on the deep learning model of this invention, which uses a ResNet50 neural network model, a 256*256 matrix block method, and a DeepLab v3+ semantic segmentation network model, shows the grading time of 20 kumquat image samples as shown in Table 3 below.

[0161] Table 3

[0162]

[0163] As can be seen from Table 3 above, it took a total of 9.08 seconds to classify 20 kumquat image samples. The average time for classifying each kumquat image sample was approximately 0.454 seconds. Compared with the traditional average classification time of 0.65 seconds per kumquat image, the segmented classification method of this invention greatly shortens the classification time and significantly improves the classification efficiency.

[0164] In addition, 150 samples of each type of kumquat were tested, and the test results are shown in Table 4 below.

[0165] Table 4

[0166]

[0167] As can be seen from Table 4 above, when using the grading method of the present invention to grade the quality of fruits and vegetables, the average accuracy rate is 98.53%, which has a relatively good grading effect.

[0168] This invention presents a fruit and vegetable grading method based on a deep learning model. On one hand, in terms of image annotation, a block-based strategy simplifies the annotation process to the image block level, significantly reducing the workload and cost. On the other hand, in terms of grading strategy, a segmented grading approach enables adaptive allocation of computational resources (neural network model and semantic segmentation network model), reducing computational workload while making the entire grading calculation smoother. This invention, from image annotation to grading strategy, reduces the overall grading workload by orders of magnitude, significantly improving work efficiency.

[0169] Furthermore, the fruit and vegetable grading method based on deep learning models in this invention divides fruit and vegetable images into blocks and labels them as a whole when training the neural network model to label the samples. The neural network model only needs to focus on learning the relatively simple task of "whether there are defects in the block image". This is easier to train and more robust than having a semantic segmentation network model learn to locate and classify all dense small targets at the same time. The accuracy of neural network model training is also higher, so that the grading accuracy of fruit and vegetable images is relatively high under the same sample set and computing budget.

[0170] For those skilled in the art, without departing from the concept of this invention, several simple deductions or substitutions can be made, and all such deductions or substitutions should be considered to fall within the scope of protection of this invention.

Claims

1. A fruit and vegetable grading method based on a deep learning model, characterized in that: Includes the following steps: C1. Real-time acquisition of fruit and vegetable images; C2. Preprocess the fruit and vegetable images c acquired in real time in step C1. The preprocessing operation includes image segmentation: extracting the fruit and vegetable parts of interest in the image and removing the background parts that are not useful for image recognition. C3. Cut the image preprocessed in step C2 into small pieces to obtain several small piece images c; C4. Using a neural network model, identify each slice image c obtained in step C4; C5. Based on the recognition results of step C4, determine whether one or more sliced ​​images c contain fruit and vegetable defects: If none of the sliced ​​images c determined in step C5 have fruit and vegetable defects, then the feature values ​​of the fruits and vegetables in the real-time fruit and vegetable images c acquired in step C1 are calculated, and the fruits and vegetables are graded according to the feature values. The feature values ​​of fruits and vegetables include one or more of the following: the area ratio of each color of the fruit and vegetable, fruit shape, and volume. If one or more of the sliced ​​images c identified in step C5 contain fruit and vegetable defects, then the semantic segmentation network model is used to identify the fruit and vegetable defects in the preprocessed fruit and vegetable images c after step C2 and determine the type of fruit and vegetable defects. After determining the type of fruit and vegetable defects, the fruits and vegetables are first graded according to the type of fruit and vegetable defects. If the fruit and vegetable defects cannot be graded by type, then the feature values ​​of the fruits and vegetables in the real-time acquired fruit and vegetable images c in step C1 are calculated, and the fruits and vegetables are graded according to the feature values. The neural network model in step C4 is a deep learning neural network model obtained after training and learning; the steps of training and learning the neural network model include: A1. Batch acquisition of fruit and vegetable images a; A2. Sample labeling; specific steps include: A2.

1. Preprocess the fruit and vegetable image a obtained in step A1. The preprocessing operation includes image segmentation: extracting the fruit and vegetable parts of interest in the image and removing the background parts that are not useful for image recognition. A2.

2. Cut the image after preprocessing in step A2.1 into small pieces to obtain several small piece images a, and divide the piece images a into training set samples; A2.

3. Perform overall annotation on the sliced ​​images a of the training set samples divided in step A2.2, where: Image 'a', which contains no fruit or vegetable defects and no black spots, is labeled as Class I. Image 'a' of a cut piece of fruit or vegetable with defects is labeled as category 2. Image 'a' of a diced fruit or vegetable without defects but with black spots is labeled as category 3. An image with one or more gray patches is considered an image with black spots; conversely, an image without any gray patches is considered an image without black spots. An image with one or more of the following fruit and vegetable defects is considered an image with fruit and vegetable defects; conversely, an image without any of these defects is considered an image without fruit and vegetable defects. A3. Neural network model training and learning; specific steps include: A3.

1. Select a neural network model; A3.

2. Using the sliced ​​image a of the training set samples labeled in step A2.3, train the neural network model determined in step A3.1; The final result is a deep learning neural network model. 2.The fruit and vegetable grading method based on a deep learning model according to claim 1, wherein: In step C2, the preprocessing operations all include binarization and noise reduction. Image segmentation, binarization, and noise reduction are performed sequentially.

3. The fruit and vegetable grading method based on a deep learning model according to claim 1, characterized in that: In the training and learning of the neural network model, the step of cutting the image after preprocessing in step A2.1 into small pieces in step A2.2 is as follows: taking the center point of the image as the center, the image after preprocessing in step A2.1 is divided into fan-shaped segments; each segmented image a has fruit and vegetable parts and background parts, and the fruit and vegetable parts are all located in the central corner region of the segmented image a, while the background part is located outside the segmented image a. Each segmented image a has the same and uniform characteristics.

4. The fruit and vegetable grading method based on a deep learning model according to claim 1, characterized in that: The semantic segmentation network model in step C5 is a deep learning semantic segmentation network model obtained after training and learning. The steps involved in training and learning a semantic segmentation network model include: B1. Batch acquisition of fruit and vegetable images; B2. Sample labeling; specific steps include: B2.

1. Preprocess the fruit and vegetable images b collected in step B1; divide the preprocessed fruit and vegetable images b into training set samples; B2.

2. Classify and label the fruit and vegetable defects in the fruit and vegetable images b in the training set samples after preprocessing in step B2.1, and classify the fruit and vegetable defects into scabs, wrinkles, punctures or bruises; B3. Semantic segmentation network model training and learning; specific steps include: B3.

1. Select a semantic segmentation network model; B3.

2. Using the fruit and vegetable images b of the training set samples labeled in step B2.2, train the semantic segmentation network model determined in step B3.1; The final result is a deep learning semantic segmentation network model.

5. The fruit and vegetable grading method based on a deep learning model according to claim 1, characterized in that: In step C5, the feature values ​​of fruits and vegetables include the area proportion of each color. The steps to calculate the area proportion of each color are as follows: convert the fruit and vegetable image c acquired in real time in step C1 from the RGB color space to the HSV color space; define the hue range of each color according to the characteristics of the HSV color space; use the defined color range to perform threshold segmentation on the image and extract the pixel region of each color respectively; calculate the area proportion of each color.

6. The fruit and vegetable grading method based on a deep learning model according to claim 1, characterized in that: In step C5, the feature values ​​of fruits and vegetables include their shape. The steps for calculating the shape of fruits and vegetables are as follows: the fruit and vegetable image c acquired in real time in step C1 is converted to grayscale; the contour of the fruits and vegetables is extracted using image segmentation technology, which can be either threshold segmentation or edge detection; the contour of the fruits and vegetables is fitted using an ellipse fitting algorithm to obtain the parameters of the ellipse; and the shape of the fruits and vegetables is determined based on the parameters of the ellipse.

7. The fruit and vegetable grading method based on a deep learning model according to claim 1, characterized in that: In step C5, the feature values ​​of fruits and vegetables include their volume. The steps for calculating the volume of fruits and vegetables are as follows: the fruit and vegetable image c acquired in real time in step C1 is converted to grayscale; the contour of the fruits and vegetables is extracted using image segmentation technology, which can be either threshold segmentation or edge detection; the contour of the fruits and vegetables is fitted using an ellipse fitting algorithm to obtain the parameters of the ellipse; and the volume of the fruits and vegetables is calculated based on the parameters of the ellipse.

8. The fruit and vegetable grading method based on a deep learning model according to claim 1, characterized in that: In step C5, the characteristic values ​​of fruits and vegetables include the area ratio of each color, fruit shape, and volume; the types of fruit and vegetable defects are divided into scabs, wrinkles, punctures, and bruises.

9. The fruit and vegetable grading method based on a deep learning model according to claim 1, characterized in that: Step C5 involves determining, based on the recognition results of step C4, whether one or more sliced ​​images c contain fruit or vegetable defects. If none of the sliced ​​images c judged in step C5 have fruit and vegetable defects, then calculate the feature values ​​of the fruit and vegetable images c collected in real time in step C1, and classify the fruit and vegetables according to the feature values: if the proportion of yellow area in each color of the fruit and vegetable is less than N% and the ratio of the long axis to the short axis of the fruit and vegetable shape is greater than M%, and the fruit and vegetable volume is either large or extra-large, then the fruit and vegetable is judged as extra-grade fruit. If the yellow area accounts for less than N% of the total area of ​​the fruit or vegetable, and the ratio of the long axis to the short axis of the fruit or vegetable is greater than M%, and the fruit or vegetable is of medium, small, or extra-small size, then the fruit or vegetable is classified as Grade 1. If the conditions of yellow area accounting for less than N% of the total area of ​​the fruit or vegetable, and the ratio of the long axis to the short axis of the fruit or vegetable is greater than M%, and the fruit or vegetable is of medium size, then the fruit or vegetable is classified as Grade 2. If the conditions of yellow area accounting for less than N% of the total area of ​​the fruit or vegetable, and the ratio of the long axis to the short axis of the fruit or vegetable is greater than M%, and the fruit or vegetable is of small size, then the fruit or vegetable is classified as Grade 3. If the conditions of yellow area accounting for less than N% of the total area of ​​the fruit or vegetable, and the ratio of the long axis to the short axis of the fruit or vegetable is greater than M%, and the fruit or vegetable is of extra-small size, then the fruit or vegetable is classified as Grade 4. If one or more of the segmented images c identified in step C5 contain fruit and vegetable defects, then the semantic segmentation network model learned in step B3.2 is used to identify the fruit and vegetable defects in the real-time acquired fruit and vegetable images c in step C1 and determine the type of defect. After determining the type of defect, the fruit and vegetable are first graded according to the defect type: if the defect type includes bruising, the fruit and vegetable are classified as grade four; if the defect type does not include bruising, but includes either puncture or wrinkling, the fruit and vegetable are classified as grade three; if the defect type does not include bruising, puncture, or wrinkling, the fruit and vegetable are classified as grade three. Then, calculate the feature values ​​of the fruits and vegetables in the real-time acquired fruit and vegetable image c in step C1, and classify the fruits and vegetables according to the feature values. At this time, if the fruit and vegetable volume is either small or extra small, the fruit and vegetable are judged as grade 3 fruits; if the fruit and vegetable volume is either medium, large or extra large, but does not meet the requirements that the proportion of yellow area in each color of the fruit and vegetable is less than N% and the ratio of the long axis to the short axis of the fruit and vegetable shape is greater than M%, the fruit and vegetable are judged as grade 2 fruits; if the fruit and vegetable volume is either medium, large or extra large, and the proportion of yellow area in each color of the fruit and vegetable is less than N% and the ratio of the long axis to the short axis of the fruit and vegetable shape is greater than M%, the fruit and vegetable are judged as grade 1 fruits. N and M are natural numbers greater than 0 and less than 100.

Images